I've just about finished this week's skit. We have a little section on flappingcrane.com where we test out videos -- we can look at them as if they were posted, though they're not. Basically so that I can be sure that everything is in place -- the video, the HTML file, the pic, etc.
However, I've noticed lately that google actually crawls our invisible skits. I assumed that someone in the group was viewing the skit, then going to google. Google would notice the referrer and crawl it.
However, this is not the case.
I put this up and viewed it. I did nothing with google. But, lo and behold, google crawled it immediately.
My best guess is it's because of the google ads. While we don't actually display google ads on those pages (what's the point?), we do have the javascript there. But it just displays one of our ads. They occasionally are displayed on the site in place of PSAs.
I don't think there's anything insidious going on here -- it's just google indexing the page so that it can place appropriate ads. Still, it was rather surprising. It's also irritating that, in theory, the invisible skit could be found on google. Not a big deal, but still irritating.
I could probably fix it by making it so none of the hidden pages communicate with google at all. But that would add unnecessary complexity to the pages, and they're complicated enough already.|||112856788774584622|||Google Crawling Invisible Skits?
cczernia|||Couldn't you just put in a robot metatag or .txt file and tell it not to index it. Joe|||Nah -- like I said, the pages are kind of complicated as is. If I were to use a meta tag, I'd have to either re-do the way that I retrieve information about the video before displaying it or I'd have to cache the page before displaying it.And a robots.txt file would be out of control. I already have to update a bunch of files to get a skit up. I really need to automate this more. But I'm the only one who has to deal with it, so I've been lazy about it.