Daft Punk of the Web

This week Daft Punk, the French electronic music duo, announced that after 28 years in the music industry they were to split.

They did this via a video called Epilogue on their YouTube channel.

But how does Daft Punk relate to the Web?

Robots rock

Daft Punk, were commonly known via their Robot personas. Similarly websites also need a Robot to be discoverable.

To be more precise a robots.txt. This is a text file that lives at the root of a website that tells machines what to look for on the site and what not to look for.

In this file we tell the bots of the web what they are allowed and disallowed from seeing or indexing (storing for them to display).

By default with no settings the bots will just look at everything in your site. For areas you don't want them to go add a Disallow rule and then for other bots you can then Allow those areas back, see my example below. ^†

These bots are referred to as User-Agents and we can target all of them together with a wildcard * (asterisk), or naming them individually.

Here is the robots.txt for this site, below I shall explain what is going on.

User-agent: Twitterbot
Allow: /images/

User-agent: facebookexternalhit
Allow: /images/

User-agent: *
Sitemap: https://code-red.uk/sitemap.xml
Disallow: /images/
    

First of all I am telling the Twitter and Facebook's bot that it is OK to go ahead and look at all the files in the /images/ folder.

Secondly I m using the * (wildcard) to target every bot and tell them where my sitemap.xml is located. I then Disallow them all from indexing all files in the /images/ folder. A sitemap is just a list of all the pages, in your site, you want the bots to look at.

Humans after all

Daft Punk, were of course not robots, they were Humans after all. Just like websites are not created by robots they are created by humans.

We can add a humans.txt file to the root to show who are the real people that have worked on, contributed or helped on a site.

While there is no requirement or standard for creating a humans.txt, http://humanstxt.org/ has some guidelines on what you might include. For example:

• Team
• Technologies
• Thanks

Some companies will use it to advertise jobs too.

Below you can see the humans.txt file for this site.

# humanstxt.org
/* OWNER & DEVELOPER */

    Name:    Dave Letorey
    Email:   dave@code-red.uk
    GitHub:  https://github.com/dletorey
    Twitter: https://twitter.com/dletorey

/* SITE */

    URL:           https://code-red.uk
    Language:      en
    Doctype:       HTML5
    Fonts:         Helvetica Neue, Georgia, Lucida Console
    Compiled with: Eleventy
    IDE:           VS Code

/* LANGUAGES */

    HTML
    CSS (SCSS)
    JavaScript
    JSON
    Nunjucks
    XML
    YAML


                                
/* THANKS */
            
    CSS Skills: Chris Burnell
    URL: https://chrisburnell.com 
    Twitter: @iamchrisburnell

    Eleventy From Scratch: Andy Bell
    URL: https://piccalil.li
    Course URL: https://piccalil.li/course/learn-eleventy-from-scratch
    Twitter: @piccalilli_
    

Conclusion

Daft Punk were an amazing part of music history and their music has not gone away, they just won't be making anymore. I don't think this is that bad a thing as they'd been dormant for a while and perhaps now they will make more music on solo projects.

In order to make your site discoverable, you'll need a robots.txt file.

The bots/user-agents that I've been talking about are Search Engines (https://www.google.com/, DuckDuckGo, Yahoo, etc) or Social Networks. These are the most common, but there are more too.

In order to show appreciation for those working on your site you should include a humans.txt file.

Listen to the tracks that inspired this article

• Epilogue - Daft Punk
• Robot Rock - Daft Punk
• Human After All - Daft Punk

† Thanks

Thanks to Alistair Shepherd for helping me clarify the default behaviour.