Control indexing per lander with a deliberate public or private status, a host-specific robots policy, page-level directives, canonical URLs, and a sitemap that contains only the URLs you want discovered.
Define intent for every host
Mark each domain and subdomain as public, staging, archived, or private. A public comparison lander should expose useful pages to crawlers; a staging clone should be kept out of search and out of public navigation. Record the decision with the domain owner so a later clone or redesign does not silently inherit the wrong policy.
Write robots.txt for the host
Serve a robots.txt file from the exact scheme and host that will be crawled. Use a specific User-agent group, Allow the public paths, and Disallow only paths that should not be crawled. Add the absolute sitemap location for a public lander. Remember that robots rules guide crawlers; they do not protect an unpublished page from someone who already knows its URL.
Use page-level directives
For a staging or temporary page, send a noindex,follow directive through the response header or HTML metadata. Remove that directive before a public launch and check the final response rather than trusting a template variable. A canonical link should point to the one preferred public URL, not to a staging host, a tracking parameter, or a different lander with different intent.
Keep sitemaps and redirects coherent
Include only canonical, reachable, indexable URLs in each lander's sitemap. Exclude redirects, error pages, duplicate parameter variants, and pages blocked by a noindex directive. After a domain move or clone, check that the old and new hosts do not create a loop and that every important page resolves to the intended HTTPS URL. Use server logs and search-console inspection to confirm how crawlers actually saw the policy.
Review each release
Make robots and canonical checks part of staging promotion. Test the public and staging host names, inspect response headers, and verify that a clone did not copy a production sitemap or analytics identifier. The hosting guide covers staging clones and domain isolation, while the speed guide helps connect crawlable pages with a usable loading path.