Tag: Tool evaluation
All blog posts with this tag.

- 05 Oct, 2026
Evaluate an analytics tool in seven days: a multi-site pilot
Most analytics comparisons focus on features, screenshots and prices. They rarely answer the questions an organisation encounters after setup. Does an action reach the report once? Can two colleagues understand the same numbers? Can an agency be restricted to one property and then removed without disrupting the rest? For a European SMB or B2B SaaS business operating multiple websites, those questions are not edge cases. They are the daily operating model. A useful trial should reproduce that model before making a purchasing decision. This seven-day pilot uses two contrasting web properties and three roles: a marketing owner, a technical implementer and a governance reviewer. It is a functional evaluation, not a seven-day experiment proving marketing performance. Low traffic may be sufficient to test instrumentation and permissions, but rarely to estimate conversion uplift reliably. Before day one: define what success means Choose two different properties, such as a lead-generation website and a documentation hub. Write down the questions each is supposed to answer. On the marketing site, the team wants to understand entry pages and completed inquiries. On the documentation property, it wants to identify useful content and navigation issues without treating every page view as a sales lead. Select three decisions the reporting should support, appoint the people responsible for testing, and record the current setup. Inventory tags, important events, URL parameters, access permissions, retention settings and the current reporting definitions. Our analytics migration checklist offers a fuller inventory when a switch has already been decided. Create a four-column test register: test case, expected result, observed evidence, difference. Evidence can be a network request, a reproducible event, an export or an access-denied response. Avoid replacing this with a subjective product score. Day 1: inspect what the browser actually sends Install the candidate on a test property or a limited live surface. Use browser developer tools to inspect requests after a page view. Look for the receiving domains, submitted fields, persistent identifiers, full URLs and query parameters. Repeat with a fictional campaign URL and with a page that should not expose sensitive details. Do not put real personal information into the test. The purpose is to establish what the product sends by default and what administrators can configure or suppress. A claim such as "cookieless" is not a legal conclusion. The French CNIL explains that consent exemptions for narrow audience-measurement purposes depend on specific purposes and implementation conditions. Ask the provider for a usable collection summary and, when relevant, its legal rationale. Evidence: a request inventory and an explanation of the configuration that produced each value. If the vendor cannot explain the flow, record a documentation gap even if the dashboard looks polished. Day 2: replay a known journey Create a short journey: land on one page, visit another, click a primary action, attempt a form submission, complete a test submission and navigate back. Repeat the sequence with timestamps. Define in advance which operations should result in page views, actions and conversions. Watch for missing events as well as double counting. A form rejected by the server should not be treated as a completed business inquiry. A button click is a useful interaction signal, but it is not itself a qualified lead. Single-page applications deserve special attention. Browser navigation can change the displayed route without loading another document. Google's official guidance recommends checking page-view events and referrer values during SPA navigation, while warning that overlapping tracking mechanisms can produce duplicates. Evidence: one reproducible table matching expected actions to recorded events, including explicit statements of what is not measured. Day 3: test definitions instead of insisting on identical totals Ask two readers to define "visit", "visitor", "page view", "event", "entry page" and "conversion". Compare their answers with the product documentation. Different tools can produce different counts because they measure different things. An absolute mismatch is not automatically a defect. Record timezone behaviour, filtering of automated requests, the consequences of blocked browser requests and any threshold or masking behaviour when a cohort is small. A vendor that explains uncertainty and unavailability can be more dependable than one that displays every number without qualification. Perfect parity with the previous platform is the wrong acceptance criterion. Reproducible definitions and understandable differences are more useful. Evidence: a short shared metrics dictionary, including an important column: "what this metric cannot establish". Day 4: test permissions across two properties Invite a colleague responsible for the marketing site and an external contractor who should only access documentation reporting. Verify visibility, configuration rights, exports and administration separately. Attempt to open the property that each role should not see. Then revoke the contractor's access and test again. Shared dashboard access is not the same as controlled permissions. Access may inherit from workspace, organisation or group settings, so the visible role name alone is not enough. The CNIL's security guidance recommends limiting permissions to work needs, removing privileges when they are no longer justified and reviewing grants regularly. Evidence: a role-by-property matrix and recorded invite and revoke tests. Treat public report links and downloaded exports as separate exposure channels. Day 5: let a non-technical owner produce the report Give the marketing owner fifteen minutes to answer three questions without help from the implementer: which pages received meaningful entrances, which areas need investigation and which useful actions are observable? Ask for a one-page meeting summary. Pay attention to how filters, date ranges, definitions and sharing behave. A minimal-looking interface may still create substantial manual work. Conversely, a sophisticated interface may be appropriate if its depth solves a genuine problem. Use the structure described in our monthly web reporting guide: one decision, a compact set of indicators, an explicit limitation and a next check. Evidence: the actual report and a record of points requiring external help. A sales demonstration cannot substitute for this independent exercise. Day 6: inspect exit, retention and cost Export what the plan actually allows. Verify column names, units, filters, time range and any omitted details. Ask what happens on account deletion, how retention periods are configured and where backups fit into the process. Calculate cost for the full target portfolio: sites, events, traffic, users, external access, required exports, APIs, support and administration effort. Providers can meter different dimensions. Plausible, for example, documents how page views and custom events count toward team-level usage. Use three scenarios rather than a false-precision forecast: ordinary demand, peak demand and growth in property count. Include switching cost and staff time. Evidence: a sourced cost model and an exit checklist that someone other than the original implementer could execute. Day 7: make a decision with non-negotiable gates Bring all three reviewers together. Mark each item as demonstrated, acceptable with a bounded follow-up, or not demonstrated. Assign ownership to follow-up actions. Do not let an attractive average score hide a critical failure: cross-property isolation, unexplained collection or irreversible access may be blocking issues.Dimension Decision question Minimum evidenceCollection Do we know what is sent and why? Observed requests and vendor documentationReliability Are known actions recorded correctly? Test log and reportDefinitions Can colleagues agree on what metrics mean? Shared dictionaryGovernance Do invite, scope and revoke work? Permission testsUsage Can a decision be made independently? Self-produced reportExit Can data be exported and deletion explained? Verified procedureCost Is multi-site use predictable? Documented scenariosA go means core requirements have been demonstrated. A conditional go means specific gaps can be closed within an agreed scope. A no-go means an essential requirement remains unproven. None of those labels is a blanket statement of legal compliance. Where Pomelo fits in such a trial Pomelo is designed around audience measurement, limited data collection and readable reporting for teams and multiple websites. That may make it worth evaluating where those priorities matter. The actual available setup, event handling, access model and commercial limits must still be tested rather than inferred from positioning. A privacy-first vendor should not win automatically. It must demonstrate functional reliability. Equally, a more complex tool may be the right answer where advanced analysis is genuinely needed. What a failed trial should produce A trial is useful even when the tool is rejected. Do not let the team leave with only a score and a preference. The useful output is a list of requirements that were demonstrated, tests that could not be run and operational assumptions that turned out to be false. For example, if two properties cannot be isolated to the intended access scopes, record the exact role, route and export attempt that revealed the problem. If a conversion appears twice, include the original user action and the network evidence. These details make a later retest much more efficient than a note saying "permissions were confusing" or "metrics looked wrong." Separate remediation from acceptance. A provider may explain that a configuration option was missed. Retest the claimed solution with the same scenario before changing the result. Conversely, avoid holding the vendor responsible for an event that your own application never emitted. Finally, retain the decision register for the next renewal cycle. The business needs and team structure can change. A previously rejected product may become relevant, while an accepted one can become unsuitable when the portfolio expands. Conclusion: test decisions, not screenshots Seven days can reveal meaningful technical and organisational evidence. It cannot always establish marketing performance. Keeping those goals separate avoids misleading confidence. The final deliverable is a concise evidence pack: collection, journeys, definitions, permissions, reporting, exit and cost. It remains useful even if the decision is to keep the current stack. FAQ Can this pilot work with very low traffic? Yes, for validating requests, events, access and exports. No, not for reliably estimating a change in conversion rate. Business-impact testing needs adequate volume and observation time. Must every site be installed during the trial? No. Two contrasting properties usually expose important configuration and governance issues before a full rollout. Does cookieless measurement automatically avoid consent requirements? No. Storage mechanisms are only part of the picture. Purpose, transmitted information, implementation and the applicable legal framework also matter. Should the new platform match GA4 counts exactly? Not necessarily. Compare definitions and reproducible journeys before assuming a difference is a defect. SourcesCNIL, audience measurement tools and consent conditions CNIL, managing permissions Google for Developers, measuring single-page applications Plausible, subscription plans and usage MDN, working with the History API
Just the right measure, for all your websites.
Pomelo is opening gradually. Join the first teams measuring right and help shape what comes next.
