Why Content Effort Matters to Google and How to Measure It

For SEOs today, publishing more content is not the problem. The challenge is knowing whether what we publish deserves to exist. Both Google's Search Quality Rater Guidelines and the leaked Google API documentation offer clues about how it might be measured. That documentation included  "contentEffort": (LLM-based effort estimation) and "ugcDiscussionEffortScore": UGC (user-generated content) page quality signals.

For years, we have measured editorial production using easy-to-count indicators: number of articles, words published, frequency, rankings, or traffic. These metrics are useful, but they do not answer a far more uncomfortable question: how much intellectual and editorial work does this page actually contribute?

Two pieces can be the same length, follow sound SEO structure, and be impeccably written. Yet one may simply rearrange what already appears in ten Google results, while the other adds original tests, calculations, a useful comparison, or a perspective grounded in real experience. They may look similar from the outside, but they are not the same for the reader.

A few days ago, I was reading a post by Cyrus Shepard on this subject, and it gave me the idea of building something that could assess or measure "Content Effort" in a way similar to how Google might do it.

That is the problem I have tried to solve with this project: turning Content Effort into something observable, debatable, and measurable, regardless of whether a perfect formula or secret Google score exists. So I created a harness with a three-judge system, a versioned measurement contract (Rubric 2.0.0), semantic contribution analysis using embeddings, and deterministic scripts.

Google Has No Public Metric Called Content Effort

This is the right place to start because SEO has a habit of turning recommendations into supposed ranking factors.

Google has not published a metric called Content Effort. Nor is there documentation stating that a page with a particular score will rank better. What does exist is a fairly clear direction of travel.

In its documentation on helpful, reliable, people-first content, Google suggests checking whether a page provides original information, research, or analysis; whether it offers more than a rewrite of other sources; whether it demonstrates first-hand expertise; and whether it delivers substantial value compared with other search results.

The guidance on AI-generated content is even more explicit: generating many pages without adding value may violate policies against scaled content abuse. The documentation itself points to the rater guidelines for identifying main content created with little effort, little originality, and little added value.

This does NOT mean human raters decide which page ranks. Google makes clear that their ratings are used to evaluate how well its systems work and do not directly influence rankings. Nor does it mean our rubric reproduces an internal Google algorithm.

It means something more useful: originality, editorial work, experience, and incremental value matter enough that we need a serious way to audit them.

What I Mean by Content Effort

To me, Content Effort is the verifiable work that turns available information into something genuinely useful to the user.

A 3,000-word article can be five sources padded into an overlong essay. This is not about measuring how long the author took, either: someone can spend two days writing a disorganized piece that contributes absolutely nothing. Nor is it about automatically rewarding human-written content over AI-generated content.

What I am trying to observe is the outcome of the work:

• what was selected and what was removed
• what data was calculated or compared
• which facts or perspectives are original
• what resources had to be created
• whether the important answer appears soon enough
• whether limitations and alternatives are explained
• whether the content connects facts that are normally presented separately
• whether there is evidence of first-hand experience

AI can play a role in the process. The criterion remains the same: does the result add knowledge, reduce uncertainty, or help someone make a decision? Automating the writing does not remove the need to do that work.

How We Built Rubric 2.0.0

I did not want to build yet another SEO traffic-light tool that returned 82 out of 100 from a checklist without explaining where the score came from. That is why "Rubric 2.0.0" works as a versioned measurement contract. It was developed through training on a human-selected set of URLs representing high-quality, low-quality, and neutral content. That selection is subjective, but subjectivity is probably part of what we are trying to measure.

Each dimension has a definition, a scale from 0 to 4, and anchors describing what must be observed to award each score. A judge cannot simply say that content "seems in-depth." It must quote the text directly and explain why those passages meet the criterion.

These are the eight intrinsic dimensions:

Each contributes between 0 and 4 points. The intrinsic result is expressed as a score out of 32 points.

This distinction matters because an article can be superbly edited yet contain no original research. Another may offer a highly valuable test but bury it behind an endless introduction. A single number enables comparison; the dimensions make improvement possible.

As an example, the dimension “3. Original facts and perspectives” looks for signs that the page contributes more than generic information or a rephrasing of a single source. Applied to a recent article on my blog, the assessment would look something like this:

Its anchors are:

Simplified Example

Level 0

“SEO helps improve visibility.”

This is a generic statement.

Level 1

“Traffic increased by 37%.”

It is specific, but we do not know where the figure came from.

Level 2

“According to the 2025 report, traffic increased by 37%.”

It has a source, but merely repeats it.

Level 3

“When we compared the 37% growth with the decline in conversions, we found that the new traffic had lower commercial intent.”

This is an original, data-supported interpretation.

Level 4

The article develops several distinct conclusions, for example about traffic, conversion, and retention, or sustains an original thesis throughout the piece by connecting specific evidence.

How an Analysis Works

The process begins before any model is called. First, we download the page and extract its main content. We save the snapshot, the extracted text, and their hashes. This tells us exactly which version was assessed and allows us to repeat the analysis without depending on the URL remaining unchanged the next day.

We then prepare a request containing the exact rubric version, the page text, the response schema, and the identifiers linking the assessment to that snapshot.

The same request is sent to three isolated judges. In the configuration we used, all three run on gpt-5.6-sol with high reasoning effort. They cannot see one another's responses and do not negotiate a shared score.

Each judge returns strict JSON containing:

1. A score from 0 to 4 for each dimension
2. The verbatim quotations supporting that score
3. A rationale tied to the rubric anchors
4. The total assessment score
5. The provenance of the model, protocol, and reasoning level

The response then passes through deterministic validators designed to catch hallucinations and errors. The schema must be correct, all eight dimensions must appear in the expected order, and every quotation must actually exist in the extracted text. If the evidence for a dimension cannot be verified, its score is voided. We do not accept a persuasive explanation unless the page content supports it.

Finally, we calculate the median of the three judges for each dimension and record the minimum, maximum, and range. We also retain the median of their individual totals. When the judges disagree, the report turns that disagreement into a reviewable signal.

Why We Use Three Judges Instead of One

A language model is not a mathematical function. It may interpret an example differently, give more weight to a piece of evidence, or draw the line between a 2 and a 3 elsewhere.

Using three judges does not eliminate that variability, but it avoids relying on a single reading. The median reduces the impact of an extreme assessment, while the range reveals where the contract still leaves room for differing interpretations.

The key lies not only in the number of judges, but in the combination of controls:

• Isolation between assessments
• The same contract for everyone
• Structured output
• Verbatim evidence
• Automated validation
• Explicit aggregation
• Calibration against reference content.

The model provides semantic judgment. The rubric, validators, and calibration prevent that judgment from becoming an unauditable opinion.

Dimension 9: Measuring Information Gain

The first eight dimensions analyze the content itself, but one question crucial to Google was still missing: what does this page contribute that is not already covered by the results competing for the same query?

That is the purpose of dimension 9. Information gain is not an absolute property of a text. It depends on a query, country, language, device, and point in time. An explanation may be novel for one SERP and completely redundant for another. The process begins with those explicit parameters. 

We request Google's organic top 10 from DataForSEO and save the response as an immutable snapshot. We then download and extract the competitors' main content. The candidate page and the top-10 pages are split into semantic chunks. We generate their embeddings using BGE-M3 running locally through LM Studio. For each query-relevant chunk, we find the semantically closest competitor content.

In practical terms, we measure two things: whether the chunk helps answer the query and how far it differs from the best coverage already available in the SERP. The results are aggregated using calibrated thresholds to produce a score from 0 to 4.

A different but irrelevant paragraph does not count as novel. Nor is changing the wording while preserving the same idea enough. We are looking for relevant information that expands, corrects, connects, or makes more concrete what the other results already offer.

Dimension 9 is presented separately from the intrinsic total. We do not turn 32 points into 36 because that would mix two different contracts: the observable quality of the content and its novelty relative to a specific SERP. A page can be extremely well constructed yet add little new information, or it can be novel but poorly edited. We need to see both.

How Much Each API Analysis Would Cost

I started building this harness with Claude, but ultimately finished it with ChatGPT. Anthropic's model dragged its feet on certain issues and was "far too keen" for me to run the judges through its API, with the additional cost that would entail. To calculate the price, I used artifacts from a real project analysis. Across the three judges, approximately 29,826 input tokens were sent and 5,579 visible output tokens were received.

The official GPT-5.6 Sol pricing, checked on August 25, 2026, is $4 per million input tokens and $20 per million output tokens. Using those figures:

The full minimum would therefore be about $0.233 per page. However, high reasoning also generates internal reasoning tokens billed as output, and those tokens were not recorded in the manual artifacts from that analysis.

Because the figure can vary, here are three possible scenarios and their costs:

DataForSEO lists a price of $0.002 for one live organic SERP with ten results. BGE-M3 embeddings run locally, so they add no per-token charge. I have not included electricity, proxies, storage, or maintenance because the cost is almost negligible.

That puts the approximate cost range at €0.20 to €0.46 per analysis.

API vs. ChatGPT Plus or Pro

AI providers' plans allow supervised analyses to run with Codex within their usage limits, avoiding marginal API charges while included capacity remains. According to the official pricing documentation, ChatGPT Plus starts at $20 per month and Pro at $100, with higher limits for Pro and the option to purchase additional credits.

If we use reference budgets of €20 and €100 per month and compare them with the range above, we get the following direct equivalent. In practice, the plans are considerably more generous, so you can probably run five to ten times as many analyses:

This does not mean Plus guarantees one hundred analyses or Pro guarantees five hundred. Each assessment requires at least three judge runs, limits are shared with other uses, and consumption depends on the model, content length, and reasoning effort. 

If I were developing a similar system for a company or launching it as a professional tool, my rule would be simple:

• ChatGPT Plus or Pro for supervised analyses, editorial research, and moderate volumes within the included capacity.
• API for automation, batch processing, CMS integration, precise telemetry, and predictable per-run costs.

The plan reduces the marginal cost when it is already part of our workflow. The API also gives you operational control.

Case Study: Analyzing the Content Effort of My Recent Post About Sistrix's Move to OAuth2

How the Result Is Calculated

1. Three independent judges receive the same page and Rubric 2.0.0.
2. They cannot consult the other judges' responses.
3. Every positive score must include verbatim quotations from the page.
4. The system verifies that those quotations actually exist; unverifiable evidence voids the corresponding score.
5. The median of the three judges is calculated for each dimension.

The judges agreed completely in this analysis:

• Totals: 26, 26, and 26
• Range: 0
• Verified evidence: 101/101
• Result: 26/32

Explanation by Dimension

The Most Influential Evidence

The judges particularly highlighted passages from the post such as:

• “In my test, the endpoint responded to initialization with HTTP 200 and text/event-stream.”
• “A decline in visibility indicates that something has changed. On its own, it does not explain the cause.”
• “The absence of results does not automatically mean zero searches, either.”
• “OAuth does not eliminate every risk.”
• “That control matters more than automating everything.”

These are not mere instructions: they demonstrate direct testing, interpretation, limitations, and reasoning, precisely what dimensions 3, 6, 7, and 8 reward.

Key Takeaways:

1. The main loss of points comes not from a lack of depth, but from editorial duplication and the absence of sophisticated media.
2. To exceed 26/32, the legacy content would need to be pruned or separated, and non-trivial informational resources added, such as an authentication-flow diagram or a structured comparison table.

Measurement Is Not Score Optimization

The obvious temptation would be to turn the rubric into another checklist: add a table to raise D4, insert an opinion to improve D3, or extend a discussion to create the appearance of depth.

That would repeat the same mistake we made with so many SEO metrics. The goal is not to decorate content for the evaluator, but to identify the work that is missing.

If D2 is low, we may not need more text; we may need to work with the data and perform analyses and calculations the reader can use. If D3 is low, an original test may be missing. If D6 is low, we are probably presenting a conclusion without acknowledging its limitations. And if D9 is low, we need to ask whether the page adds anything new or merely competes by rewording the same information.

The tool does not predict rankings, either, because a page's search ranking depends on relevance, authority, links, context, user experience, and many other factors. What it does is distinguish content that looks complete from content that demonstrates genuine effort.

That is the real value of this system: automating part of the assessment so that human judgment can focus on what matters. Not publishing faster, but understanding what deserves to be published and what we should improve before asking for the reader's time.