Quadrants, waves, maps, spider charts, and now AI-generated landscapes. The production method changed four times. The unit of analysis never changed once.
There is an argument running in this space about whether AI can replace the analyst in producing market maps.
It is the wrong argument, and I want to show you why with two records rather than an opinion.
I built the best version of the thing I am about to criticize
In 2005 my company was financed by the Business Development Bank of Canada. That financing followed research funded under the Government of Canada’s Scientific Research and Experimental Development program — self-learning algorithms in a nascent AI environment, beginning in 1998. Without the SR&ED work there would have been no BDC round.
As a condition of the financing, BDC required an independent market SWOT.
In September 2005 an independent firm — no relationship with the vendors it assessed, no inclusion fees, no participants to keep happy — produced a 105-page Strategic Market Assessment of the e-procurement software industry. It named 23 entries as the principal players. None exceeded $50 million in revenue in 2004.
My company commissioned it and paid for it. I hold the original.
That last part is why this exercise is possible at all. A subscription research instrument cannot be reproduced, so nobody can take one and follow it forward in public. I can. I have — publishing the record in 2011, again in 2024, and now at the twenty-one year mark. Three dated publication points, each recording what was true when it was written.
Read what it shows. Fifteen of the twenty-three named entries are now associated with organizations that lost independent ownership. One had already been absorbed before the report appeared — B2eMarkets ends before the report line, because it was inside VerticalNet when the document was bound. Seven more followed within three years. Silver Oak ends almost exactly on the line. The market was consolidating while the assessment was being written.
That is a vendor map, produced independently, followed forward for two decades, with every confirmed corporate event sourced and every unresolved trail shown as unresolved. It is the most rigorous version of this instrument I have seen, and I paid for it.
It tells a buyer nothing about whether anything worked.
The second record
[GRAPH 2 — THE FLOOR THAT DID NOT MOVE]
A separate record, built from unrelated sources, covering roughly the same period.
The rate moved. It went from 84% down to 63%. The floor did not. Across twenty-six years the proportion of projects that did not fully succeed never fell below 63 percent.
⚠ These two graphs are not connected and I am not connecting them. Vendor churn and implementation outcomes are different phenomena. An acquisition can be an excellent outcome for a vendor and entirely irrelevant to a buyer’s implementation. Neither record was built from the other. They are two separately established facts that happen to cover the same thirty years.
What both records point at
Every instrument in this family shares one axis.
Quadrants position vendors. Waves score vendors. Maps place vendors. Spider charts cluster vendors. Landscapes categorize vendors. The 2005 assessment I commissioned assessed vendors.
The production method has changed at least four times in thirty years — analyst desk research, then survey panels, then crowd-sourced reference data, and now large-scale content processing. Each generation is faster than the last and covers more of the market.
The unit of analysis has never changed once. It has always been the vendor.
No recurring instrument in this family uses the buyer’s implementation as its unit of analysis.
That is not an editorial preference or a complaint about anyone’s methodology. It is a structural observation about an entire category of market intelligence: for three decades it has been measuring the supplier, at the moment of selection, in a market where the outcome is determined afterward and somewhere else.
Before anyone overstates this
The vendor is not nothing. Capability is real. Viability is real. Whether a supplier will still exist in three years is a legitimate question, and Graph 1 is the answer to it for one 2005 cohort.
But vendor identity alone is not sufficient to determine the outcome.
Virginia and North Carolina implemented the same platform and produced opposite results. The supplier variable was shared. The operating conditions were not.
Whatever decided it was not a property of the vendor, because the vendor was the same in both cases.
What I am actually arguing
We should stop covering the vendors and start covering the outcomes.
Not because vendor coverage is inherently inaccurate. It is mostly careful, and the people producing it are serious. Because we have been treating vendor coverage as outcome coverage, and it is not. It answers a question that is settled at signature, and leaves untouched the question that decides everything after it.
An instrument whose unit was the implementation would ask different things. What did this organization believe it was buying? What operating condition turned out to matter that nobody scoped? Who could change the frame once it was running? How long did they keep the vendor, the product direction, the service model and the commercial relationship they thought they had selected? And the one nobody publishes: did it work?
None of those questions has a vendor as its subject.
Thirty years, four production methods, one axis. Whether the next map is drawn by an analyst or generated by a model is a question about how fast we can keep measuring the wrong object.
Truth Is Believing. Accuracy Is Knowing. Outcome Is Proof.™
Thirty Years of Measuring the Wrong Object
Posted on September 29, 2026
0
Quadrants, waves, maps, spider charts, and now AI-generated landscapes. The production method changed four times. The unit of analysis never changed once.
There is an argument running in this space about whether AI can replace the analyst in producing market maps.
It is the wrong argument, and I want to show you why with two records rather than an opinion.
I built the best version of the thing I am about to criticize
In 2005 my company was financed by the Business Development Bank of Canada. That financing followed research funded under the Government of Canada’s Scientific Research and Experimental Development program — self-learning algorithms in a nascent AI environment, beginning in 1998. Without the SR&ED work there would have been no BDC round.
As a condition of the financing, BDC required an independent market SWOT.
In September 2005 an independent firm — no relationship with the vendors it assessed, no inclusion fees, no participants to keep happy — produced a 105-page Strategic Market Assessment of the e-procurement software industry. It named 23 entries as the principal players. None exceeded $50 million in revenue in 2004.
My company commissioned it and paid for it. I hold the original.
That last part is why this exercise is possible at all. A subscription research instrument cannot be reproduced, so nobody can take one and follow it forward in public. I can. I have — publishing the record in 2011, again in 2024, and now at the twenty-one year mark. Three dated publication points, each recording what was true when it was written.
Read what it shows. Fifteen of the twenty-three named entries are now associated with organizations that lost independent ownership. One had already been absorbed before the report appeared — B2eMarkets ends before the report line, because it was inside VerticalNet when the document was bound. Seven more followed within three years. Silver Oak ends almost exactly on the line. The market was consolidating while the assessment was being written.
That is a vendor map, produced independently, followed forward for two decades, with every confirmed corporate event sourced and every unresolved trail shown as unresolved. It is the most rigorous version of this instrument I have seen, and I paid for it.
It tells a buyer nothing about whether anything worked.
The second record
[GRAPH 2 — THE FLOOR THAT DID NOT MOVE]
A separate record, built from unrelated sources, covering roughly the same period.
The rate moved. It went from 84% down to 63%. The floor did not. Across twenty-six years the proportion of projects that did not fully succeed never fell below 63 percent.
⚠ These two graphs are not connected and I am not connecting them. Vendor churn and implementation outcomes are different phenomena. An acquisition can be an excellent outcome for a vendor and entirely irrelevant to a buyer’s implementation. Neither record was built from the other. They are two separately established facts that happen to cover the same thirty years.
What both records point at
Every instrument in this family shares one axis.
Quadrants position vendors. Waves score vendors. Maps place vendors. Spider charts cluster vendors. Landscapes categorize vendors. The 2005 assessment I commissioned assessed vendors.
The production method has changed at least four times in thirty years — analyst desk research, then survey panels, then crowd-sourced reference data, and now large-scale content processing. Each generation is faster than the last and covers more of the market.
The unit of analysis has never changed once. It has always been the vendor.
No recurring instrument in this family uses the buyer’s implementation as its unit of analysis.
That is not an editorial preference or a complaint about anyone’s methodology. It is a structural observation about an entire category of market intelligence: for three decades it has been measuring the supplier, at the moment of selection, in a market where the outcome is determined afterward and somewhere else.
Before anyone overstates this
The vendor is not nothing. Capability is real. Viability is real. Whether a supplier will still exist in three years is a legitimate question, and Graph 1 is the answer to it for one 2005 cohort.
But vendor identity alone is not sufficient to determine the outcome.
Virginia and North Carolina implemented the same platform and produced opposite results. The supplier variable was shared. The operating conditions were not.
Whatever decided it was not a property of the vendor, because the vendor was the same in both cases.
What I am actually arguing
We should stop covering the vendors and start covering the outcomes.
Not because vendor coverage is inherently inaccurate. It is mostly careful, and the people producing it are serious. Because we have been treating vendor coverage as outcome coverage, and it is not. It answers a question that is settled at signature, and leaves untouched the question that decides everything after it.
An instrument whose unit was the implementation would ask different things. What did this organization believe it was buying? What operating condition turned out to matter that nobody scoped? Who could change the frame once it was running? How long did they keep the vendor, the product direction, the service model and the commercial relationship they thought they had selected? And the one nobody publishes: did it work?
None of those questions has a vendor as its subject.
Thirty years, four production methods, one axis. Whether the next map is drawn by an analyst or generated by a model is a question about how fast we can keep measuring the wrong object.
Truth Is Believing. Accuracy Is Knowing. Outcome Is Proof.™
-30-
Share this:
Like this:
Related