{"id":139,"date":"2006-01-10T17:44:28","date_gmt":"2006-01-11T00:44:28","guid":{"rendered":"http:\/\/www.pdxtc.com\/wpblog\/archives\/139"},"modified":"2014-12-04T13:54:44","modified_gmt":"2014-12-04T20:54:44","slug":"what-does-a-search-engine-spider-see-at-your-website","status":"publish","type":"post","link":"https:\/\/www.pdxtc.com\/wpblog\/search-engines\/what-does-a-search-engine-spider-see-at-your-website\/","title":{"rendered":"What does a search engine spider see at your website?"},"content":{"rendered":"

Did you ever wonder what pages of your website get “indexed” or “spidered” by a search engine? Improper design can make some of your pages invisible to the search engines. <\/p>\n

John Alexander, of Search Engine Workshops<\/a> brought this Search Engine Spider simulator<\/a> to my attention in his “SEO tip of the day”. <\/p>\n

Just type in your URL, and you’ll get a list of every page the spider can get to. Are some of your pages completely missing? Find out here<\/a>.<\/p>\n","protected":false},"excerpt":{"rendered":"

Did you ever wonder what pages of your website get “indexed” or “spidered” by a search engine? Improper design can make some of your pages invisible to the search engines. John Alexander, of Search Engine Workshops brought this Search Engine Spider simulator to my attention in his “SEO tip of the day”. Just type in […]<\/p>\n","protected":false},"author":76,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_et_pb_use_builder":"","_et_pb_old_content":"","_et_gb_content_width":"","footnotes":""},"categories":[7,16],"tags":[],"class_list":["post-139","post","type-post","status-publish","format-standard","hentry","category-search-engines","category-seo"],"yoast_head":"\nWhat does a search engine spider see at your website?<\/title>\n<meta name=\"description\" content=\"Did you ever wonder what pages of your website get "indexed" or "spidered" by a search engine? Improper design can make some of your pages invisible to\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.pdxtc.com\/wpblog\/search-engines\/what-does-a-search-engine-spider-see-at-your-website\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"What does a search engine spider see at your website?\" \/>\n<meta property=\"og:description\" content=\"Did you ever wonder what pages of your website get "indexed" or "spidered" by a search engine? Improper design can make some of your pages invisible to\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.pdxtc.com\/wpblog\/search-engines\/what-does-a-search-engine-spider-see-at-your-website\/\" \/>\n<meta property=\"og:site_name\" content=\"Scott Hendison's Old Search Commander Blog\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/SearchCommander\" \/>\n<meta property=\"article:published_time\" content=\"2006-01-11T00:44:28+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2014-12-04T20:54:44+00:00\" \/>\n<meta name=\"author\" content=\"Scott\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@shendison\" \/>\n<meta name=\"twitter:site\" content=\"@shendison\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Scott\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\\\/\\\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\\\/\\\/www.pdxtc.com\\\/wpblog\\\/search-engines\\\/what-does-a-search-engine-spider-see-at-your-website\\\/#article\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/www.pdxtc.com\\\/wpblog\\\/search-engines\\\/what-does-a-search-engine-spider-see-at-your-website\\\/\"},\"author\":{\"name\":\"Scott\",\"@id\":\"https:\\\/\\\/www.pdxtc.com\\\/wpblog\\\/#\\\/schema\\\/person\\\/3142c7d28dc676725ac62cd6c9de8371\"},\"headline\":\"What does a search engine spider see at your website?\",\"datePublished\":\"2006-01-11T00:44:28+00:00\",\"dateModified\":\"2014-12-04T20:54:44+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\\\/\\\/www.pdxtc.com\\\/wpblog\\\/search-engines\\\/what-does-a-search-engine-spider-see-at-your-website\\\/\"},\"wordCount\":90,\"commentCount\":0,\"articleSection\":[\"Search Engines\",\"SEO\"],\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"CommentAction\",\"name\":\"Comment\",\"target\":[\"https:\\\/\\\/www.pdxtc.com\\\/wpblog\\\/search-engines\\\/what-does-a-search-engine-spider-see-at-your-website\\\/#respond\"]}]},{\"@type\":\"WebPage\",\"@id\":\"https:\\\/\\\/www.pdxtc.com\\\/wpblog\\\/search-engines\\\/what-does-a-search-engine-spider-see-at-your-website\\\/\",\"url\":\"https:\\\/\\\/www.pdxtc.com\\\/wpblog\\\/search-engines\\\/what-does-a-search-engine-spider-see-at-your-website\\\/\",\"name\":\"What does a search engine spider see at your website?\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/www.pdxtc.com\\\/wpblog\\\/#website\"},\"datePublished\":\"2006-01-11T00:44:28+00:00\",\"dateModified\":\"2014-12-04T20:54:44+00:00\",\"author\":{\"@id\":\"https:\\\/\\\/www.pdxtc.com\\\/wpblog\\\/#\\\/schema\\\/person\\\/3142c7d28dc676725ac62cd6c9de8371\"},\"description\":\"Did you ever wonder what pages of your website get \\\"indexed\\\" or \\\"spidered\\\" by a search engine? Improper design can make some of your pages invisible to\",\"breadcrumb\":{\"@id\":\"https:\\\/\\\/www.pdxtc.com\\\/wpblog\\\/search-engines\\\/what-does-a-search-engine-spider-see-at-your-website\\\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\\\/\\\/www.pdxtc.com\\\/wpblog\\\/search-engines\\\/what-does-a-search-engine-spider-see-at-your-website\\\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\\\/\\\/www.pdxtc.com\\\/wpblog\\\/search-engines\\\/what-does-a-search-engine-spider-see-at-your-website\\\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\\\/\\\/www.pdxtc.com\\\/wpblog\\\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"What does a search engine spider see at your website?\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\\\/\\\/www.pdxtc.com\\\/wpblog\\\/#website\",\"url\":\"https:\\\/\\\/www.pdxtc.com\\\/wpblog\\\/\",\"name\":\"Scott Hendison's Old Search Commander Blog\",\"description\":\"\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\\\/\\\/www.pdxtc.com\\\/wpblog\\\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Person\",\"@id\":\"https:\\\/\\\/www.pdxtc.com\\\/wpblog\\\/#\\\/schema\\\/person\\\/3142c7d28dc676725ac62cd6c9de8371\",\"name\":\"Scott\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/ba275e23c0aad37526141e715b54cd3eeac27b071e4395b2b39e801ca68355d6?s=96&r=g\",\"url\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/ba275e23c0aad37526141e715b54cd3eeac27b071e4395b2b39e801ca68355d6?s=96&r=g\",\"contentUrl\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/ba275e23c0aad37526141e715b54cd3eeac27b071e4395b2b39e801ca68355d6?s=96&r=g\",\"caption\":\"Scott\"},\"sameAs\":[\"https:\\\/\\\/x.com\\\/shendison\"]}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"What does a search engine spider see at your website?","description":"Did you ever wonder what pages of your website get \"indexed\" or \"spidered\" by a search engine? Improper design can make some of your pages invisible to","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.pdxtc.com\/wpblog\/search-engines\/what-does-a-search-engine-spider-see-at-your-website\/","og_locale":"en_US","og_type":"article","og_title":"What does a search engine spider see at your website?","og_description":"Did you ever wonder what pages of your website get \"indexed\" or \"spidered\" by a search engine? Improper design can make some of your pages invisible to","og_url":"https:\/\/www.pdxtc.com\/wpblog\/search-engines\/what-does-a-search-engine-spider-see-at-your-website\/","og_site_name":"Scott Hendison's Old Search Commander Blog","article_publisher":"https:\/\/www.facebook.com\/SearchCommander","article_published_time":"2006-01-11T00:44:28+00:00","article_modified_time":"2014-12-04T20:54:44+00:00","author":"Scott","twitter_card":"summary_large_image","twitter_creator":"@shendison","twitter_site":"@shendison","twitter_misc":{"Written by":"Scott"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/www.pdxtc.com\/wpblog\/search-engines\/what-does-a-search-engine-spider-see-at-your-website\/#article","isPartOf":{"@id":"https:\/\/www.pdxtc.com\/wpblog\/search-engines\/what-does-a-search-engine-spider-see-at-your-website\/"},"author":{"name":"Scott","@id":"https:\/\/www.pdxtc.com\/wpblog\/#\/schema\/person\/3142c7d28dc676725ac62cd6c9de8371"},"headline":"What does a search engine spider see at your website?","datePublished":"2006-01-11T00:44:28+00:00","dateModified":"2014-12-04T20:54:44+00:00","mainEntityOfPage":{"@id":"https:\/\/www.pdxtc.com\/wpblog\/search-engines\/what-does-a-search-engine-spider-see-at-your-website\/"},"wordCount":90,"commentCount":0,"articleSection":["Search Engines","SEO"],"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/www.pdxtc.com\/wpblog\/search-engines\/what-does-a-search-engine-spider-see-at-your-website\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/www.pdxtc.com\/wpblog\/search-engines\/what-does-a-search-engine-spider-see-at-your-website\/","url":"https:\/\/www.pdxtc.com\/wpblog\/search-engines\/what-does-a-search-engine-spider-see-at-your-website\/","name":"What does a search engine spider see at your website?","isPartOf":{"@id":"https:\/\/www.pdxtc.com\/wpblog\/#website"},"datePublished":"2006-01-11T00:44:28+00:00","dateModified":"2014-12-04T20:54:44+00:00","author":{"@id":"https:\/\/www.pdxtc.com\/wpblog\/#\/schema\/person\/3142c7d28dc676725ac62cd6c9de8371"},"description":"Did you ever wonder what pages of your website get \"indexed\" or \"spidered\" by a search engine? Improper design can make some of your pages invisible to","breadcrumb":{"@id":"https:\/\/www.pdxtc.com\/wpblog\/search-engines\/what-does-a-search-engine-spider-see-at-your-website\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.pdxtc.com\/wpblog\/search-engines\/what-does-a-search-engine-spider-see-at-your-website\/"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/www.pdxtc.com\/wpblog\/search-engines\/what-does-a-search-engine-spider-see-at-your-website\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/www.pdxtc.com\/wpblog\/"},{"@type":"ListItem","position":2,"name":"What does a search engine spider see at your website?"}]},{"@type":"WebSite","@id":"https:\/\/www.pdxtc.com\/wpblog\/#website","url":"https:\/\/www.pdxtc.com\/wpblog\/","name":"Scott Hendison's Old Search Commander Blog","description":"","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.pdxtc.com\/wpblog\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Person","@id":"https:\/\/www.pdxtc.com\/wpblog\/#\/schema\/person\/3142c7d28dc676725ac62cd6c9de8371","name":"Scott","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/secure.gravatar.com\/avatar\/ba275e23c0aad37526141e715b54cd3eeac27b071e4395b2b39e801ca68355d6?s=96&r=g","url":"https:\/\/secure.gravatar.com\/avatar\/ba275e23c0aad37526141e715b54cd3eeac27b071e4395b2b39e801ca68355d6?s=96&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/ba275e23c0aad37526141e715b54cd3eeac27b071e4395b2b39e801ca68355d6?s=96&r=g","caption":"Scott"},"sameAs":["https:\/\/x.com\/shendison"]}]}},"_links":{"self":[{"href":"https:\/\/www.pdxtc.com\/wpblog\/wp-json\/wp\/v2\/posts\/139","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.pdxtc.com\/wpblog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.pdxtc.com\/wpblog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.pdxtc.com\/wpblog\/wp-json\/wp\/v2\/users\/76"}],"replies":[{"embeddable":true,"href":"https:\/\/www.pdxtc.com\/wpblog\/wp-json\/wp\/v2\/comments?post=139"}],"version-history":[{"count":0,"href":"https:\/\/www.pdxtc.com\/wpblog\/wp-json\/wp\/v2\/posts\/139\/revisions"}],"wp:attachment":[{"href":"https:\/\/www.pdxtc.com\/wpblog\/wp-json\/wp\/v2\/media?parent=139"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.pdxtc.com\/wpblog\/wp-json\/wp\/v2\/categories?post=139"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.pdxtc.com\/wpblog\/wp-json\/wp\/v2\/tags?post=139"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}