SemiAnalysis Gross Mafia Tactics?

[Daniel Nenni]

Interesting perspective on SemiAnalysis tactics from one of my LinkedIn connections.

I am sure you will delete this, so I will screenshot it and post it after you do. It shows you don’t understand how keyword advertising works. They don’t buy ads. They bid on keywords and the whole process is automated. The fact they shunned you is exemplary. They don’t want to be any part of your gross mafia tactics. Kudos to IBM!!! Oh and congrats on admitting to fraud.

Jon Stevens​

CEO Hot Aisle - AMD AI Developer Cloud

https://www.linkedin.com/in/jon-s-stevens

SemiAnalysisSemiAnalysis 4 hours ago • Visible to anyone on or off LinkedIn

Interestingly, IBM Cloud has started buying ads based on the "clustermax" keyword. This is strange, since IBM actively worked against us during our testing period.

During our intro call with IBM, the analyst relations team stated that they "would only be willing to participate if guaranteed a silver rating or higher".

Maybe this is how it works with analysts like Gartner, and why their ratings show IBM > Nebius and CoreWeave, and OVHcloud > Lambda

In terms of technology, when we asked to test a slurm or kubernetes cluster (or both) IBM Product Management teams spent time trying to convince us to use their scheduler LSF instead of Slurm, OpenShift instead of Kubernetes, and Spectrum Scale (GPFS) instead of Weka, VAST, or Lustre.

After being denied access as an analyst, we tried to register for an account directly as a customer.

This led to an IBM Account Verification team calling our engineers on their personal cell phones, interrogating what we were using their services for, blocking our payments, and eventually shut off our IBM cloud account completely.

With all that said, it did take us only 45 seconds to spin up a new machine, a little bit longer to assign a floating IP, and access it.

IBM does not pre-install NVIDIA drivers, docker, and the nvidia container toolkit in their base image, and that's about all we've been able to figure out so far.

IBM currently has a Bronze rating since clusters seem hard to come by. We look forward to properly testing their services in the future.

For more stories like this, check out the full ClusterMAX 2.0 article on our newsletter.

https://www.linkedin.com/in/jon-s-stevens

 

[swka]

Interesting perspective on SemiAnalysis tactics from one of my LinkedIn connections.

I am sure you will delete this, so I will screenshot it and post it after you do. It shows you don’t understand how keyword advertising works. They don’t buy ads. They bid on keywords and the whole process is automated. The fact they shunned you is exemplary. They don’t want to be any part of your gross mafia tactics. Kudos to IBM!!! Oh and congrats on admitting to fraud.

Jon Stevens​

CEO Hot Aisle - AMD AI Developer Cloud

https://www.linkedin.com/in/jon-s-stevens

SemiAnalysisSemiAnalysis 4 hours ago • Visible to anyone on or off LinkedIn

Interestingly, IBM Cloud has started buying ads based on the "clustermax" keyword. This is strange, since IBM actively worked against us during our testing period.

During our intro call with IBM, the analyst relations team stated that they "would only be willing to participate if guaranteed a silver rating or higher".

Maybe this is how it works with analysts like Gartner, and why their ratings show IBM > Nebius and CoreWeave, and OVHcloud > Lambda

In terms of technology, when we asked to test a slurm or kubernetes cluster (or both) IBM Product Management teams spent time trying to convince us to use their scheduler LSF instead of Slurm, OpenShift instead of Kubernetes, and Spectrum Scale (GPFS) instead of Weka, VAST, or Lustre.

After being denied access as an analyst, we tried to register for an account directly as a customer.

This led to an IBM Account Verification team calling our engineers on their personal cell phones, interrogating what we were using their services for, blocking our payments, and eventually shut off our IBM cloud account completely.

With all that said, it did take us only 45 seconds to spin up a new machine, a little bit longer to assign a floating IP, and access it.

IBM does not pre-install NVIDIA drivers, docker, and the nvidia container toolkit in their base image, and that's about all we've been able to figure out so far.

IBM currently has a Bronze rating since clusters seem hard to come by. We look forward to properly testing their services in the future.

For more stories like this, check out the full ClusterMAX 2.0 article on our newsletter.

https://www.linkedin.com/in/jon-s-stevens

so the fraud is how semianalysis gained access to IBM server/service when they cannot. On the other hand, if IBM's rating is on silver ad above, this will not likely to surface.

what are the actual technical facts regarding their servers?

 

[Daniel Nenni]

so the fraud is how semianalysis gained access to IBM server/service when they cannot. On the other hand, if IBM's rating is on silver ad above, this will not likely to surface.

what are the actual technical facts regarding their servers?

You probably have to pay SemiAnalysis for that information. This is way out of my lane but I do question the business practice. IBM clearly said SemiAnalysis was not allowed to run their benchmarks on the IBM servers and SemiAnalysis did it anyway. This could have legal repercussions so I am a bit surprised they are bragging about it. I have delt with IBM on a number of occasions and found them to be overly litigious. I have been to court on a number of occasions and found it to be a colossal waste of time and money, especially against a company like IBM. Just my opinion of course.

 

[swka]

You probably have to pay SemiAnalysis for that information. This is way out of my lane but I do question the business practice. IBM clearly said SemiAnalysis was not allowed to run their benchmarks on the IBM servers and SemiAnalysis did it anyway. This could have legal repercussions so I am a bit surprised they are bragging about it. I have delt with IBM on a number of occasions and found them to be overly litigious. I have been to court on a number of occasions and found it to be a colossal waste of time and money, especially against a company like IBM. Just my opinion of course.

agree.

 

[Xebec]

On the flip side, can you trust Clustermax's results when the testing is done with the cloud provider knowing that "the benchmark" is going to rate them and influence how much business they may get. It becomes trivial for the cloud provider to flag the "Clustermax" account for extra performance, elevated priority, availability, etc.

 

[Daniel Nenni]

On the flip side, can you trust Clustermax's results when the testing is done with the cloud provider knowing that "the benchmark" is going to rate them and influence how much business they may get. It becomes trivial for the cloud provider to flag the "Clustermax" account for extra performance, elevated priority, availability, etc.

I did benchmarks for a three letter computer company as an intern back in the day. They were highly manipulated so I have never trusted them since. I prefer the term "mileage may very..." because it certainly does.

SemiAnalysis did in fact delete his comment so Jon Stevens was right about that.

 

[delong.height]

Kinda wild how SemiAnalysis is perpetually beefing with the AMD fanboys.

 

[KevinK]

This raises the question for me of who the real villain is:
* A benchmarking company that gives grades for realistic AI neocloud players and realistic performance curves for data center hardware, but requires the company they are benchmarking to provide access to the hardware, plus some level of tech support ?
* Or a neocloud or hardware company that wants to be benchmarked, but will only agree to the testing if they are guaranteed a good grade (IBM’s approach) ?

If I’m a buyer of neocloud services or hardware, the second one is the bad actor.

Kinda wild how SemiAnalysis is perpetually beefing with the AMD fanboys.

From what I have seen, AMD has invested a lot of senior software and system R&D bandwidth in collaborating with SemiAnalysis to achieve far better at slot level inference (no optimized rack level results to date). So if I were AMD, I might actually be a little upset about one of my developer partners whining about being benchmarked - makes them look bad.

I believe that the AI data center markets are paying far more attention to SemiAnalysis results, vs the other guys, Artificial Analysis because SemiAnalysis does far more realistic data center scale benchmarking with reasonable, though still not the perfect simulated workloads. Artificial Analysis is now scrambling to develop and offer similar.