Procurement

Of 674 AI tool vendors' terms read, 315 restrict benchmarking or competitor use

The author, who runs a site that grades tools used by AI agents, says that of 674 vendor terms they read, 315 restrict benchmarking or using the service to build a competitor, 107 name benchmarking, 69 ban it and 28 require permission to publish results. They name LangSmith and Coinbase as examples and say Cursor allows publishing if others can repeat the test. The counts are theirs and unaudited.

Why it matters If you plan to run your own evals on a vendor's API, check its terms first, since the author says many forbid publishing results.