This week Daft Punk, the French electronic music duo, announced that after 28 years in the music industry they were to split.
They did this via a video called Epilogue on their YouTube channel.
But how does Daft Punk relate to the Web?
Daft Punk, were commonly known via their Robot personas. Similarly websites also need a Robot to be discoverable.
To be more precise a
robots.txt. This is a text file that lives at the root of a website that tells machines what to look for on the site and what not to look for.
In this file we tell the bots of the web what they are allowed and disallowed from seeing or indexing (storing for them to display).
By default with no settings the bots will just look at everything in your site. For areas you don't want them to go add a
Disallow rule and then for other bots you can then
Allow those areas back, see my example below. †
These bots are referred to as User-Agents and we can target all of them together with a wildcard
* (asterisk), or naming them individually.
Here is the robots.txt for this site, below I shall explain what is going on.
User-agent: Twitterbot Allow: /images/ User-agent: facebookexternalhit Allow: /images/ User-agent: * Sitemap: https://code-red.uk/sitemap.xml Disallow: /images/
First of all I am telling the Twitter and Facebook's bot that it is OK to go ahead and look at all the files in the
Secondly I m using the
* (wildcard) to target every bot and tell them where my
sitemap.xml is located. I then
Disallow them all from indexing all files in the
/images/ folder. A sitemap is just a list of all the pages, in your site, you want the bots to look at.
Humans after all
Daft Punk, were of course not robots, they were Humans after all. Just like websites are not created by robots they are created by humans.
We can add a
humans.txt file to the root to show who are the real people that have worked on, contributed or helped on a site.
While there is no requirement or standard for creating a
humans.txt, http://humanstxt.org/ has some guidelines on what you might include. For example:
Some companies will use it to advertise jobs too.
Below you can see the humans.txt file for this site.
Daft Punk were an amazing part of music history and their music has not gone away, they just won't be making anymore. I don't think this is that bad a thing as they'd been dormant for a while and perhaps now they will make more music on solo projects.
In order to make your site discoverable, you'll need a
The bots/user-agents that I've been talking about are Search Engines (https://www.google.com/, DuckDuckGo, Yahoo, etc) or Social Networks. These are the most common, but there are more too.
In order to show appreciation for those working on your site you should include a
Listen to the tracks that inspired this article
Thanks to Alistair Shepherd for helping me clarify the default behaviour.
Other articles from the blog
=, == or ===