Member of the Month

hi_vis360
November, 2024

Online

Welcome to the Newschoolers forums! You may read the forums as a guest, however you must be a registered member to post. Register to become a member today!

Anyone have experience with server/render farms/multiple machine setups?

profile picture
1337
Posts: 31059 - Karma: 29,329
This idea came to me when I was looking at new computer parts. I have a decent computer now (i7 960 3.2 Quad, 12gb RAM) but my gpu is shitting the bed on me, and my motherboard/processor seem to hang unnecessarily long at startup which worries me. It's two years old and has done quite a bit and I'm content with buying a new mobo/cpu/gpu and stealing old parts to build a new beast. But why not use it until it dies? So I started to research how to wire two computers together to split rendering processes/creating a render farm. I had a hard time finding an answer for that direct question of just using an old cpu tethered to a new one, most suggested building from a few identical and new machines.

So a few questions I have are:
-would I be better off buying one new system (i7 3770k based, ~1,250) and wiring the old computer to it and hope I can split the tasks between the two?
-Is it possible to do what i'm thinking with two different computers, aka just using one as a bitch renderer to take up a bit of the work to speed up the process? What software/hardware setups would I need?
-would I be better buying a few cheap quad core server setups (2-4 depending on configuration, would like to keep the project sub 2k), replace the GPU on my current computer, and use it as the master to control the nodes?
-Links/guides/101 tutorials on how to get it all working/setup would be a huge help. Found some basic ones, would like a start to finish if possible just to see what I need. Especially when it comes to wiring all the ethernet together/creating a network. That confused me really quick.

I'd highly consider a switch to windows, but a linux based beowulf cluster is out of the question as I'd be using it for CS6 apps, cinema 4d, etc. If I could stick to mac, that'd be great, all my stuff is currently mac formatted, but i'm not sure you have access to the programs required that I found windows based.

Cheers for the help,
Jamie

PS: and because I know people will ask, "why the fuck do you want to do this/need to do this?" The answer is because I enjoy spending the time to learn about a subject I have no prior knowledge in, and figured this could be a cool way to spend some time and elbow grease configuring something that could be useful to know and practical to use for both now and future use. You can create a lot of power with not a lot of money building systems like this (mac pros are what, 5-6k for a 12 core machine? where this can give you 16 cores with an insane amount of ram for ~2k if done properly)
Oct 22 2012 11:51AM
0 0
profile picture
STAUSS
Posts: 8232 - Karma: 3,311
this shit makes little or no sense to me but "linux based beowulf cluster" would be a sick name for a metal band. 
Oct 22 2012 12:13PM
0 0
profile picture
tomPietrowski
Posts: 7162 - Karma: 17,592
I use a simply multi machine setup for rendering. Its a great way to make use of old computers which still work, but are not fast enough for work.

I use backburner from autodesk as my server client. I use my main Workstation (i7 3820, 16gb ram) as the master, and then conect the other machines as servers. I can connect them all by simply connecting all the machines to one router.

You can control the machines from he mater machine, so the servers don need there own input devices and screens, which helps money wise.

You can set it so that all the machines render to a main drive, or o there internal hard drive.

If you go for 1 drive make sure its big but also fast and has a fast connection to all machines.

For the servers work out what you need to render and get nothing more.

Graphics cards are not really needed, just enough ram to process the info you give it.

In terms of rendering, I usally spilt the scene in a number of renderable chunks and have the machines work on there own set of frames. You can have it set so all the machines work together, which is very usefull if your working on a still and want to speed it up.

For adobe, you can get a background server, I forget the name, but you can buy a number of after market control programs too.

Hope that answers some of your questions, let me know if you have any more.
Oct 22 2012 2:21PM
0 0
profile picture
Jaggy_Snake
Posts: 2287 - Karma: 1,418
Have you considered upgrading your RAM to a speed of 1600MHz, using your editing programs on your SSD (RAID them maybe), or even using 2 graphics cards?
Oct 22 2012 4:35PM
0 0
profile picture
1337
Posts: 31059 - Karma: 29,329
Ram runs at 1333mHz, edit off an SSD/Raid array. Two gpu's wouldnt do much unless i was using cuda.

The computer is showing signs of wear and tear, and I was just thinking of putting it to good use should it be replaced
Oct 22 2012 4:39PM
0 0
profile picture
tomPietrowski
Posts: 7162 - Karma: 17,592
What I would really consider is how long things are taking to render.

For example one of the scenes Im currently working on takes 90min a frame to render, so thats 36hours for 1sec of footage. So for me its worth having a reasonably fast machine to work on, but then have secondary machines to do the rendering. This way I can still work and just let the machines render in the background.

However for adobe things a full render may only be 1-2 hours. For that type of work it may be better making your workstation faster. If you only render for say 60min, its not a huge downtime on your main workstation. You will prob find for projects like that, that 1 fast workstation will be best.

What GPU are you currently using and what problems are you having. What are you looking to upgrade too, Quadro or consumer?

The main inprovment will be CPU, so maybe look into a Xeon setup if you stick with one machine.

You could always make a simple render node with the bits you upgrade.
Oct 22 2012 4:57PM
0 0
profile picture
1337
Posts: 31059 - Karma: 29,329
Current GPU has caused a few freezes and kernel panics on my machine, as well as runs very hot even at idle with my 2560x1440 monitor (it's a radeon 5770, 1gb 128bit). Temps average 60c idle and spike to 80c when running something like minecraft or shredsauce. I'm looking at the 660ti to replace it.

And that's exactly what I'm planing to do; most likely upgrade to a 3770k i7, new mobo, gpu, then take most of the missing parts from my current setup and put together a new computer. But if I can use my old one to be a render bitch, then I might as well try to do that and just get a whole new outfit for the 3770k. My questions are just how to wire it/software it all together

and when i researched that, I found it would cost the same amount to buy a few previous gen xeon or sb quad core systems and use them as a 12/16 core farm. but I probably don't need to go that far.
Oct 22 2012 5:05PM
0 0
profile picture
tomPietrowski
Posts: 7162 - Karma: 17,592
The 660 will be good overall if you only work in adobe, but if you plan on expanding into even simply 3d then a quadro may be better. The Quadro 2000 will be ok, but maybe look at the Quadro 4000.

If you go i7 over Xeon, then go for the 39?? I forget the numbers. Its the new 6 core chip, so esetially 12 threads. Xeons do cost alot but offer a dual setup so you could get 2 cheaper xeons and still have a faster machine then 1 expensive i7.

Software wise, check backburner, cube, tractor. They all offer server client rendering. As I say I use backburner. Its pretty simple but does help in customisation if you now a little C++, but you can find most of the info online.
Oct 22 2012 5:11PM
0 0
profile picture
Jaggy_Snake
Posts: 2287 - Karma: 1,418
I'd say go for it!

And make a thread on how you done it. It sounds really interesting as I've never really heard of anything like this before.
Oct 22 2012 5:12PM
0 0
profile picture
1337
Posts: 31059 - Karma: 29,329
go for the 3770k, or 3 quad core machines?
Oct 22 2012 5:13PM
0 0
profile picture
Jaggy_Snake
Posts: 2287 - Karma: 1,418
Go for "the render bitch" machine (Your current?) and build a new, then link the 2, however it's done.
Oct 22 2012 5:15PM
0 0
profile picture
1337
Posts: 31059 - Karma: 29,329
word, probably what I'll end up doing.

I'm going to get some rack mount server cases for both machines. They seem to offer better air flow, I can keep both machines close together (and maybe rack a raid setup later), and they look cool ;)

My list of hardware so far:
--Intel i7 3770k 3.5gHz Quad Core CPU
--GIGABYTE GA-Z77X-UD5H Mobo (has 2x gigabit ethernet, does it work to run the render into one and internet to the other?)
--Gigabyte 660ti 2gb 192-bit GPU
--16gb (4x4gb) G.Skill Ripjaw 1333mHz RAM (same as my other computer)
--750W Corsair PSU
--GELID Solutions Slim Silence i-Plus (CC-SSilence-iplus) 75mm Ball Bearing CPU Cooler

Case and etc:

--Norco 3u Rack Mount Case (http://www.newegg.com/Product/Product.aspx?Item=N82E16811219047)
--iStarUSA 2x5.25" Drive bay to 3x3.5" Hot Swappable drive bays (http://www.newegg.com/Product/Product.aspx?Item=N82E16816215243)
--nMEDIAPC ZE-C668 5.25" Card reader/Front Panel Audio/USB3 hub (http://www.newegg.com/Product/Product.aspx?Item=N82E16811996043)

For my render, a new housing so it can sit with this one
--ARK 3u Rack Mount Case (http://www.newegg.com/Product/Product.aspx?Item=N82E16811128065)
Oct 22 2012 5:43PM
0 0
profile picture
tomPietrowski
Posts: 7162 - Karma: 17,592
Looks like an ok setup, I would look at upgrading that CPU though. Its good for now, but will become very outdated soon. Look for the 6 or 8 core (8core are stupid expensive right now though).

For the ram, maybe consider get 2x8Gb sticks of faster ram. I did not look how many slots that Mobo has but if its 8 you could get 2 more 8Gb sticks and run 32Gb of ram. I would say thats a minimum for a i7 running after effects. AE just eats ram. Adobe recommend 2gb minimum for each core, so if you have a quad i7 with hyperthreading the minimum recommend is 16Gb.

I would also maybe look at upgrading that GPU, its ok, but you could certainly get something better, for not much more. At least look at the GTX 670, but I would look at the Quadro 2000 and 4000 too.

Regarding the 2 ethernet conections, you can run a server and the internet through the same one. If you wanted to run a seperate router not conected to the net, for the farm that would work with the 2 connections.

Thats what I do, I do not have my workstation of render nodes online, just connected to an unconnected router.
Oct 22 2012 7:41PM
0 0
profile picture
1337
Posts: 31059 - Karma: 29,329
What are the differences between the quadros and something like the gtx670/660ti?

And my ultimate goal is to get the best bang for my buck. If I'm spending $600 on a single lower end 6 core from intel, why not get three quad cores for the same price and build three machines that run together? I know there are other things to consider, but the i7 3930k (SB 3.2 6core) achieves ~150gflops and a single i5 3570 (IB 3.4 4core) hits about 108, stringing 3 of the 3570's together would get me twice as much power for the same price. No?

What processors should I be looking at if I have that wrong? What's the difference between a xeon and i7? Would going a few releases to say Westmere xeon's and trying to find a dual socket board with 2x quad xeons be better?
Oct 23 2012 1:27AM
0 0
profile picture
tomPietrowski
Posts: 7162 - Karma: 17,592
The quadros and the GTX boards are based on the same tech, were they differ is the drivers they use. The quadro drivers are set up so well for 3d and 2d work. They are heavily intergrated into adobe and autodesk products. Certainly in 3d the benchmarks are crazy, the quadros are 4-5 times faster then equal power GTX boards.

Regarding buying one expensive CPu compared to 3 lesser powered cheaper ones. Yes in multithreaded programs 3 slower ones prob would be faster, however in a non multi threaded program you may find it evens out. So look at what your working with in that respect.

The other thing to consider is Ram. If you use 32Gb on your Workstation, really you need 32Gb in all your render nodes to ensure they can process the same material your workstation can. 96Gb of ram will be about $45o so that off sets the cost of the CPU somewhat.

What I would do, is upgrade your CPU, Ram and GPU on your current build. You will prob have to upgrade the Mobo at the same time. Stick your old CPU, Ram and Mobo in a render node. Sell the GPU if you can to offset the cost of a new powersuply and case for the render node.

If you already use a raid HD setup, just point the render node to that. Use your new faster worksation as the master and the old parts in the render node as a slave. Then you can work and render at once and if you need you could set them both to render at the same time over night for example
Oct 23 2012 10:14PM
0 0
profile picture
1337
Posts: 31059 - Karma: 29,329
Well my current build is a i7 960, which runs triple channel memory. I could max it out to 24 when I move it, but wont be able to hit 32 to match a master machine.

Any suggestions for a single beast master computer? Would you suggest a 6core/processor? What about something like an EVGA SR-2 and dual xeons (I'd get two quad cores for 8 total, couldnt budget 2x$600 6cores)
Oct 24 2012 1:15AM
0 0
profile picture
Balto
Posts: 25985 - Karma: 4,404
just get a 12 core mac pro u sissy
Oct 24 2012 1:38AM
0 0
All times are Eastern (-5)