Benchmarking the AI performance of different types of AI hardware can feel like comparing apples to oranges. How can you make a fair comparison? Let's clear up the confusion.
Intro to the Problem
Remember the days when buying a new processor was all about who could crank up the GHz the most? It was a simpler time, when you could just check the clock frequency and make your decision. But then, multicore processors came along and changed the game. Suddenly, comparing different processor architectures wasn't so straightforward. It wasn't just about raw speed anymore – it was about whether the hardware could handle the specific tasks you needed it for. Could this processor run my game smoothly? Could it handle all the tracks I wanted to record in my DAW (Digital Audio Workstation)? These were the real questions driving my buying decisions.
Today, many of our customers face the same challenge when evaluating AI hardware performance. But where does the confusion come from, and what do you really need to know to make an informed buying decision for AI hardware?
TOPS: Cool or Confusion?
The biggest source of confusion around AI performance is undoubtedly the metric that all hardware manufacturers – including us – advertise: TOPs (Tera Operations per Second). This simplified metric measures how many operations AI hardware can perform in a second. However, TOPs do not differentiate between the quality or type of operation being performed. And what about the datatype of the operation – was it measured with INT8, INT16, or FP32?
Not all applications are the same. The requirements of the different tasks vary greatly.
In my opinion, TOPs have some value as a ballpark figure to understand general differences when comparing hardware with similar architectures, but not much beyond that.
Breaking it down: What you really need
But what if you want to know if the hardware is right for your AI application? The simple answer is to benchmark it using a workload that closely matches your use case. Then, check if both the quality of the outcome and the speed of inference meet your needs. This is what truly answers the question: “Will it do what I want it to do?”
Back to our CPU example: CPU benchmarking has gone through the same evolution. Today, performance is tested across multiple metrics, including single-thread and multi-thread performance, power efficiency, cache throughput, and many more. For different use cases, different metrics are more or less useful. In the AI world, we focus on model accuracy and inference time (or throughput).
Get the balance right
With more and more AI professionals looking for cost-effective alternatives to NVIDIA's expensive solutions, understanding true performance metrics is more important than ever. At the same time, the current possibilities of integrated AI accelerators for Computer-on-Modules are becoming increasingly relevant.
At congatec, we provide benchmarking results based on specific workloads and AI models, helping you make an informed decision. If your project requires benchmarking scrutiny before deployment, let's talk!
Watch also my video about the conga-TC750 with Intel Core Ultra Series 2 technology
