Technical SEO is the work of making a site's important pages accessible and understandable to search engines. Beginners should start with access, internal links, crawl directives, indexing signals, and Search Console. Google discovers URLs, crawls pages, indexes their contents, and serves results. However, Google does not guarantee that every page will be crawled, indexed, or shown, according to Google Search Central's explanation of how Search works.
Table of Contents
- Understand discovery, crawling, and indexing
- Use robots.txt and noindex correctly
- Manage sitemaps and duplicate URLs
- Diagnose problems with Search Console
- Add enhancements after access is sound
Understand discovery, crawling, and indexing
A page must pass through several stages before it can appear in search results. Discovery means google finds the URL. Crawling means it visits the page, while indexing means it processes the page's contents.
Start with the pages that matter most to visitors or the business. These might include service pages, product categories, articles, campaign landing pages, or key documentation. For each important page, check four basic conditions: These checks narrow the problem. A page Google cannot reach needs an access fix, while a reachable but non-indexed page requires closer inspection.
- The page is publicly accessible.
- Another page on the site links to it.
- Crawling is not blocked.
- The page does not contain a `noindex` directive.
Use robots.txt and noindex correctly
A `robots.txt` file controls crawler access and can help manage crawl load. It is not a dependable privacy or removal tool.
Google warns that a blocked URL may still appear in results when another website links to it, as explained in its robots.txt guidance. Use the control that matches the goal: Do not assume that blocking a page makes it confidential. Pages containing customer records, internal reports, staging material, or other sensitive information need access controls rather than crawler instructions.
- Use `robots.txt` to manage crawler access.
- Use `noindex` when a public HTML page should not be indexed.
- Use authentication when content must remain private.
Manage sitemaps and duplicate URLs
An XML sitemap gives Google a list of canonical URLs the site wants surfaced. Each entry should be a fully qualified address, such as `https://example.com/services/design`, rather than `/services/design`. A sitemap supports discovery but does not force crawling. Treat it as a clean inventory of preferred URLs, not a command or a guarantee.
Remove redirected, duplicate, or unwanted addresses from that inventory. Canonicalization tells search engines which URL represents duplicate or nearly duplicate pages. Google Search Central describes redirects and `rel="canonical"` as strong preference signals, while sitemap inclusion is weaker; Google still decides whether URLs are duplicates in its canonicalization documentation. For example, tracking parameters might create several addresses for one product page. Point those variants toward the preferred URL, link internally to that URL, and list only that version in the sitemap.
Diagnose problems with Search Console
Use the URL Inspection tool when one important page is missing or behaving unexpectedly. It shows what Google knows about that URL and can test the live page, making it the focused choice for an individual crawling or indexing problem, according to Google Search Console Help. Use the Page Indexing report for a broader view. It groups indexed and non-indexed URLs and gives reasons for exclusions.
Review unexpected exclusions first instead of trying to make every reported URL indexed. Duplicate and redirected URLs normally do not need separate index entries. A useful audit asks whether each preferred page is indexed, not whether the report shows 100% coverage. A practical first review is:.
- Inspect one high-priority URL.
- Test its live version.
- Confirm that it is accessible and indexable.
- Review the Page Indexing reason if it is excluded.
- Compare the page with its canonical URL and sitemap entry.
Add enhancements after access is sound
Structured data supplies standardized clues about a page's meaning. Valid markup can make a page eligible for rich results, but eligibility does not guarantee that Google will display them. Add structured data only after resolving access, indexing, duplication, and canonical problems. Validation confirms that markup follows the expected format; it does not override Google's display decisions.
Performance also deserves attention because Core Web Vitals measure user experience across visits. The Chrome team's targets are an LCP of 2.5 seconds or less, an INP of 200 milliseconds or less, and a CLS of 0.1 or less at the 75th percentile. Investigate real-user results rather than treating one test as the final verdict. Start with the metric that misses its target, identify the affected page type, and measure again after making a focused change.
