How We Research and Compare Software

There is a difference between a reviewer completing a task, watching a vendor demonstrate it, reading the vendor’s documentation and quoting a customer’s experience. Each can be useful evidence, but none should be presented as another kind.

The difference matters when a ranking drives a software budget, training, or a data migration. Every recommendation of that weight needs a source you can check and a reason that source applies to your task.

How the evidence framework works

The framework separates four kinds of evidence: tasks completed independently, vendor-guided demonstrations, official documentation, and attributed user reports. A good review says which kind supports each claim and which business requirement the claim covers. Without a documented firsthand test, a claim rests on the other evidence the review names.

Why we compare software by use case

A shortlist starts with the buyer’s job and the capabilities that job requires. A small service company following up with new leads needs different CRM capabilities from a sales department managing territory permissions and complex reporting.

Field-service software adds other tasks: scheduling a visit, dispatching a technician, preparing an estimate, and invoicing the customer. Project-management software raises questions about assignments, dependencies, permissions, and whether the team will adopt the workflow. One feature checklist for all three categories would hide the differences that matter.

A transparent shortlist explains three things:

• Admission: The product fits the business scenario and meets the criteria for this comparison.
• Exclusion: A confirmed limitation, category mismatch, or disqualifying requirement keeps a product out.
• Unresolved status: A potentially relevant product has too little evidence to include or reject fairly.

Plenty of ratings can make discovery harder, as one Reddit contributor said while comparing software directories:

Both of them have tons of reviews, but it can get messy, too many options, some reviews feel fake

The CMO’s published software review methodology gives an example of category-specific selection criteria and intended audiences, which explain why a product enters a particular comparison.

What counts as hands-on testing?

Hands-on testing means an evaluator uses a product to complete a defined task and records the outcome under stated conditions. The account edition, permissions, input data, and action attempted all shape what the result means.

A guided demonstration shows what a vendor presents as possible. Developer documentation describes supported features and requirements. A user report gives one person’s account of their experience.

Evidence mode	What the record supports	What should be disclosed
Independent hands-on test	Observed outcome of a specific task	Product, edition, environment, task, date, and result
Vendor-guided demonstration	Behavior shown in the vendor’s presentation	Who controlled the demo and which configuration appeared
Official documentation	The developer’s published feature or configuration claim	Document, applicable plan, and verification date
Attributed user report	A contributor’s account of a specific experience	Role, use case, relevant conditions, and source

Take a CRM evaluation: import sanitized contacts, create a sales stage, assign a follow-up, generate a report, and export customer records. Each step gets its own recorded result. A field-service test could follow one request through estimate, field assignment, job completion, and invoice.

The test record for such a workflow includes the evaluator, product and edition, date, sample input, outcome, and any feature or permission restriction. If the editor could not access the required tier, that fact goes next to the missing result.

A Hacker News participant used review directories for a narrower purpose:

I use them mainly for discovery, and do my own evaluation on the tools, as the crowdsourced reviews are mostly crap.

How we compare features, trade-offs, and scores

Fair comparisons judge products on criteria that matter to the same business problem. A CRM review can weigh contact import, follow-up, permissions, reporting, and export. A field-service review may prioritize mobile dispatch, estimates, job completion, and invoicing. Requirements that stop the core workflow from working come before optional conveniences.

Comparable tasks do not always come with equally strong evidence. Consider this hypothetical comparison between two unnamed CRM products:

Criterion	Software A	Software B	How to interpret the result
Contact import	Observed in a task test	Listed in vendor documentation	Different types of support for the same capability
Reporting	Observed on evaluated plan	Not tested	B’s outcome is unknown
Customer export	Unavailable on evaluated plan	Documented for a different edition	Plan conditions decide the comparison
Technician dispatch	Not applicable	Not applicable	Outside the defined CRM task

“Not tested,” “failed in a documented test,” and “not applicable” are different outcomes. A feature missing from one evaluated edition may be available in another.

A numeric score is useful when its calculation is defined: disclosed criteria, actual weights, a rule for unknown results, and clear disqualifiers. TrustRadius, for example, describes its trScore as a weighted rating measure. When a publisher’s formula is unknown, a comparison of specific strengths and limitations is the supported assessment. The scorecard example with a failed must-have illustrates why a failed requirement cannot be offset by optional features.

How we evaluate user reviews

User reviews can describe support interactions, setup problems, and day-to-day friction that a short evaluation might never encounter. How useful they are depends on the contributor’s role, company context, product edition, and specific experience.

Ask three questions about any outside review: who wrote it, how they were invited, and whether they received an incentive. Identity verification, invitation source, and compensation are three separate facts about a review.

TrustRadius identifies three ways reviews can be sourced:

Invitation route	Who initiates it	What the reader learns
Independent invitation	TrustRadius	The platform recruited the contributor
Platform invitation for a vendor	TrustRadius, on the vendor’s behalf	The solicitation has a vendor relationship
Vendor invitation	The vendor	Customer selection and invitation practices deserve attention

An incentive can be offered within any recruitment route. Whether a reward exists and what conditions attach to it are separate facts.

G2’s community guidelines describe automated screening and human moderation and list labels including Validated Reviewer, Current User, Incentivized, Source, and Rating Update. The labels address identity, sourcing, incentives, or revision.

In 2021, one person who said they wanted to leave a favorable review described trouble with identity checks:

it will be rejected unless you hand over a LinkedIn account or corporate email.

A vendor-side account shows how customers get invited to review software:

We’ve been running automated G2 review collection since 2018 using the standard approach of asking users to complete an in-app NPS survey

The SaaS operator said invitations targeted customers who scored 9 or 10 out of 10 on that survey. Real customers can still form a selectively recruited group, so the recruitment pathway matters when you read the resulting ratings. Before you weigh ratings, reviewers, and methodology, read how to start with the claim that matters.

The FTC’s rule FAQ also says the rule does not specifically prohibit asking only customers expected to be happy for reviews, while that practice may still violate the broader FTC Act.

Can commercial relationships affect our recommendations?

Commercial relationships change how a recommendation should be disclosed. An affiliate payment, a sponsored placement, and a vendor-provided test account each mean something different to the reader.

Relationship	What should be disclosed	Decision consequence
Affiliate commission	Compensation tied to referrals or purchases	The reader can recognize a material financial connection
Sponsored placement	Whether a position or promotional display was paid for	Paid visibility is distinguishable from an editorial assessment
Vendor-provided access	Conditions of a trial license or guided demonstration	The scope and independence of the evidence are clear

Paid directory visibility can bring a vendor to buyers’ attention, and an affiliate link can pay a publisher. In both cases, the reason for the recommendation and the commercial relationship must be stated clearly.

The CMO has published an affiliate policy describing commission-bearing and non-affiliate links and its stated separation between revenue and editorial decisions. It is the kind of policy readers can inspect.

The FTC Endorsement Guides FAQ explains that material connections affecting an endorsement need clear and conspicuous disclosure, placed in the context of the specific endorsement or affiliate link.

How review updates and corrections are documented

Software facts change: a feature moves to a higher subscription tier, integration permissions change, or an update alters what users can do. A trustworthy review records when each important claim was last checked and how a substantive correction affected the judgment.

Four dates describe different events: first publication, an editorial update, verification of a factual claim, and a hands-on retest. A last-updated field helps only when it reflects the work actually done. A cosmetic revision does not show that a feature was retested.

Suppose a CRM comparison says a reporting feature is available in an entry-level edition, and the vendor later restricts it to another plan. The correction would name the original claim, the newly applicable edition, the supporting document or observation, and the changed recommendation, if any.

A traceable correction record contains:

• Changed claim: The earlier statement and the corrected wording.
• Reason: A factual error, altered product capability, revised source, or editorial reassessment.
• Evidence: The document or reproducible observation behind the change.
• Verification date: When the affected information was checked.
• Decision effect: Whether eligibility, a comparison, or a recommendation changed.

A usable correction channel names a responsible contact and explains how reported errors are handled.

What our reviews cannot establish, and how to verify a claim

Evidence from one edition and task applies only to the conditions recorded for that evaluation. A business with a different subscription, integration, permission setting, device, or dataset can get another result. For the underlying selection method, see workflow-first software selection.

For a CRM comparison, exporting the customer records from the proposed subscription is a more revealing follow-up than relying on the review’s feature checklist.