Discussion Forum

Subtitle: COMSOL-CFD-Mixer 5.2a is AMAZING Software ! – Robust, Accurate, and Affordable

I know most blog posts stem from complaints or questions of one sort or another (including most of mine in the past), but I think a strong unsolicited independent compliment is long overdue, as least for COMSOL CFD and Mixer. (First, parenthetically I should note that my complaints in the past arose from my (very disappointing) attempts to use COMSOL RF for broadband calculations of S-parameters on extremely complex RF and microwave problems. I understand that major new capabilities can be expected within ~6 months that should make COMSOL RF better able to handle such problems. I look forward to checking that out in due course…)

But back to my current focus – CFD for Rotating Machinery with transonic flow at “moderately low” Reynolds numbers and extremely high shear stresses. One part of a major project we are working on involves the development of very small micro-turbine-driven air-bearing-supported sample spinners needed for determining molecular structures of biological macromolecules in so many crucial cases where the other methods (x-ray crystallography, cryo electron microscopy, solution NMR, mutagenesis, etc.) simply don’t work. Fast “Magic Angle Spinning (MAS-NMR)” allows NMR to determine atomic-resolution structures in the most difficult cases, such as the amyloid-beta fibrils that play a key role in Alzheimer’s Disease. One of our goals is to be spinning small samples (under 0.8 mm diameter) at over 7 million rpm within a few years, and it’s looking like COMSOL CFD Mixer will be playing a key role in helping us develop the extremely stable and robust instrument required.

We’ve been making MAS NMR instruments for a long time, and we used another CFD product for a short period about 15 years ago to help us make some advances at that time, but we needed better tools now. Our primary concern was compressible non-isothermal flow in rotating machinery at Mach numbers up to 1.8 for Reynolds numbers in the range of 500-15,000 with very high boundary layer shear stresses. This is an area few have investigated.

We started by evaluating the three leading products – product X, product Y, and COMSOL CFD (even though some “experts” had told us COMSOL CFD couldn’t be trusted to give correct answers!). Product X was the only one advertised to work for problems like ours, but we soon ruled it out, partly because it was prohibitively expensive for our purposes (we needed at least two seats), but also because it was so difficult to get useful information from their “upgraded” website, at least at that time. (That wasn’t the case a year earlier, and may not be the case now – haven’t checked.) We actually found it even much more difficult to get useful and relevant information on product Y, but we had a colleague with considerable prior experience with it so we got a trial license and ran some validations, as with COMSOL CFD.

Our validation cases included simple small transonic nozzles with sharp edges. We also looked at several micro-channel flows for which exact solutions were well known. For the micro transonic nozzles we compared the simulation results to the experimental data. In all cases, for Mach numbers up to ~1.7, the COMSOL-CFD simulation results using the L-VEL turbulence model were within experimental error (~15%). The product Y results (and we tried every turbulence model available) were not even close, and their technical support could not help! (The product Y runs were also several times slower than the COMSOL-CFD runs for similar number of mesh cells.) For the laminar flow cases we compared the COMSOL-CFD simulation results to the analytic solutions, and all were within ~1%. (We didn’t check product Y here, as the transonic micro-nozzles had already ruled it out.) Undoubtedly, product Y gives valid results for many incompressible flow cases and probably for many compressible low-velocity cases, but it clearly doesn’t begin to compare to COMSOL-CFD for tough problems.

The model we are currently working on has ~800k mesh cells, about 1000 boundaries (with boundary-layer meshes on most of them), and the meshed file size without solutions is ~100MB. (This model includes 20 transonic nozzles with abrupt expansions and a number of areas where the mesh quality is not very good.) Even though Mach numbers in many places exceed 1.5, the solution converges adequately in 30-40 iterations using the direct PARDISO solver (and the L-VEL turbulence model, etc.). We have been running it on six different modern high-end computers. Our experience may be of interest to others for computer-choice purposes.

We are often running two instances of our model simultaneously, so that is how we first chose to benchmark the following three computers, all operating under Windows 10.

Computer A, laptop, with an i7-6820HQ and 64 GB DDR4 RAM;
(4 cores, 3.5 GHz turbo, 34 GB/s 8MB cache); SS HD.
Computer B, with two E5-2640 v3’s and 128 GB DDR4 RAM;
(each: 8 cores, 3.4 GHz turbo, 59 GB/s, 20MB cache); C612; 6GB/s SSD.
Computer C, with two E5-2687 v3’s and 256 GB DDR4 RAM;
(each: 10 cores, 3.5 GHz turbo, 68 GB/s, 25MB cache); C612; 7200 rpm 1TB HD.

On computer A, this exercise (two simultaneous simulations, each 36 iterations) takes 163 minutes.
On computer B, this same exercise took ~91 minutes.
On computer C, this same test took ~76 minutes.

We then ran eight simultaneous instances of the same CFD problem on computer B and found the average run time to be 340 minutes, or about 3.7 times that for two instances. A single instance of this problem runs in 62 minutes on computer B.

We then ran a single instance of this problem on a fourth computer:

Computer D, with an i7-3930K and Windows 7; 7200 rpm HD;
(6 cores, 3.2 GHz turbo, 51 GB/s, 12MB cache, 64 GB DDR3 RAM).

This single-instance simulation took 85 minutes. Computer D, with its single (modest) processor, was so slow in an attempt to run two instances of the above problem that we abandoned the attempt (looked like it would take more than twice as long as the single instance).

From the above, it looks like memory bandwidth is the main factor. Other benchmarks of COMSOL-RF on Computer D and another computer (an i7-5960X) also showed speed for single instances to depend mostly on memory bandwidth of the processor used and clock speed.

There appears to be little benefit from having more than 6 cores in each processor, no matter how many instances of large COMSOL problems are running. There is a small benefit from dual processors (as in Computer B and Computer C) when running multiple instances of large problems (particularly in making the computer more responsive to other applications with COMSOL running in the background). However, we actually saw a substantial decrease in the speed of COMSOL-RF on a high-end four-socket machine (four D5-4627v2’s, SS HD) compared to Computer D – but maybe that was a symptom of the slow chip sets that seem to be available for quad-socket boards.

From the above, we surmise that the best current choice for demanding COMSOL problems would be a dual-socket machine with two E5-2697v4’s (each 16 cores, 3.6 GHz turbo, 77 GB/s, 40MB cache, $2900). The E7-4830v4 (each 14 cores, 2.8 GHz turbo, 102 GB/s, 35MB cache, $2170) has higher rated memory bandwidth, but our experience with quad-socket boards is very bad, and clock speed is probably as important as memory bandwidth. (Of course, if COMSOL could do a better job of effectively using more than ~6 cores, that would also be greatly appreciated by those of us needing to run thousands of huge problems over the next few years!)

For those on a tighter budget and only concerned about speed for single instances of large COMSOL problems, an i7-6950X ($1750) is probably the best choice, as long as its 128 GB RAM limit is sufficient.

Now, a shout-out for version 5.2a. After learning COMSOL-CFD on version 5.2, our initial response to 5.2a was not completely positive, as there were new bugs to learn to work around (mostly, consistently losing track of the correct boundaries in component couplings, and others, that I could list). However, the pluses definitely outweighed the minuses. From our perspective, the most important was the greatly increased robustness of the boundary-layer meshing. Now, it just works! Always. No matter how weird the domain and boundaries. And there are other new features in 5.2a that are critical to our work, most notably, the k-omega and L-VEL turbulence models with non-isothermal compressible flow for rotating machinery in the mixer module.

The above described CFD benchmarks (based on a portion of our micro MAS-NMR spinner) were run without the rotor spinning – using just the CFD module (spf and ht physics). We then moved on to the spinning case, using Rotating Machinery Non-Isothermal Turbulent Flow (rmnitf) physics.

We compared a subset of the previous problem (with a slightly coarser mesh) with a non-rotating rotor (using spf and ht physics) to the same problem with a rotating rotor (where some rotor surface speeds were ~0.9 Mach number) using rmnitf physics. The non-rotating problem had 302K mesh cells and the rotating case had 514K mesh cells (mostly because the smaller clearances from the need for the identity-pair boundary required smaller mesh elements in many places). On Computer B, the non-rotating (302K mesh) problem ran in 15.5 minutes, and the rotating problem (514K mesh) took 61 minutes. On Computer C, the non-rotating problem ran in 16 minutes (302K mesh), and the rotating problem (514K mesh) initially took a whopping 121 minutes! About 3 times longer than expected!

While we were disappointed with the poor degree of parallelization (often Task Manager was showing only 5-7% CPU utilization by COMSOL on Computer C), COMSOL-CFD-Mixer seems to be correctly solving our toughest transonic microturbine problems once everything is set up properly. We are finding that the k-omega model works accurately and robustly for Mach numbers up to 1.6 at Reynolds numbers below 10,000 even though it’s developers weren’t targeting that regime!


We then replaced the conventional 7200 rpm hard-drive in Computer C with a 32 GB/s 800GB SSD (an Intel 750 NVMe) and saw a dramatic increase in speed on large mixer problems. Now, on such problems, the upgraded Computer C (which we’ll now call Computer E) typically was 36% faster than Computer B. Surprisingly, Task Manager still showed very long stretches of 4-7% CPU utilization on every step of solver 2 (PARDISO), while Computer B had only very short interludes below 20% CPU utilization and was solidly at 50% most of the time.

At this point we have to get more computer speed (most of our single-instance runs are now more that 6 hours), so we’re going to put together another computer based on two E5-2697v4’s – and of course, a 32GB/s SSD (critical for large mixer problems) and 2400MHz DDR4 memory. We’re also going to get a computer based on an i7-6950X. We’ll report our experience with these computers at some point in the future.

Finally, COMSOL-CFD-Mixer 5.2a really is amazing software – after one gets through most of the learning curve. Perhaps the biggest deficiency is the poor training documentation and the on-line help. It’s always much too brief and obtuse, always written for those who already know all the jargon, and very seldom has links to relevant working examples that illustrate the topic. (Keep in mind that these comments are coming from an old physicist with four decades of experience in and out of fluid dynamic, including several brief stints with earlier CFD products.) If the folks at COMSOL-CFD could improve parallelization in their Mixer module and then focus on improving their on-line help and providing many more helpful examples, particularly in the area of compressible non-isothermal 3D flow in high-speed rotating machinery (where they have only one real competitor), they could totally own this area !

At some point in the future, I hope to get around to posting a “COMSOL-CFD Guide for Dummies” – the important stuff they omit from the tutorials, the Quick Starts, and even from the detailed manuals, that you really need to know!

Cheers!

F. David Doty, PhD
Doty Scientific Inc
Columbia SC