Welcome to blot’s documentation!¶

Composable Pipelining¶

What if we could ease out those internal mechanisms and make them composable? If we could, we wouldn’t be limited only to the behaviors available from built-in mechanisms. We can simply use a different mechanism that serves the same role but more how we’d like.

This results in a much more procedural build process. Instead of writing a long list of configuration variables, you’re specifying the mechanisms of the pipeline for each kind of content you have.

Site Generation Process¶

Site generation is modeled as a generic process of “content transformation”. A site may contain various kinds of content:

• chronological articles
• static pages
• structured data
• media assets (css, js, images)

For the purposes of site generation, the concerns for each of these content types are the same:

• where is the content source coming from?
• how is the content source parsed?
• how should parsed content assets be processed?
• how should output content be rendered?
• where should output content be written?

In each case the need to load some stuff from disk, maybe do stuff to it, maybe render it into something else, and to write it somewhere is universal. We can visualize the pipeline for a given content type as:

+-------------------+        +------------------+       +------------------+
|                   |        |                  |       |                  |
|  Content Sources  +------> |  Content Assets  +-----> |  Output Content  |
|                   |        |                  |       |                  |
+-------------------+        +------------------+       +------------------+

Asset Processing¶

Once content sources have been loaded from disk and potentially parsed, the resulting content assets are objects that contain the content and optional metadata. Its easy to imagine then, those assets finding their way to the disk at the end of the pipeline.

For example, we might have some articles written in Markdown, which are loaded and parsed into content assets. Those articles are then written out as html in a convienent place. But what about other kinds of content that don’t have source files on disk? Categories are good example.

If you have a category assigned in the metadata of each article, you probably want pages generated that index the articles belonging to that category. But so far, we’ve only seen that content assets that come from content sources can be written out. However, during asset processing, additional asset types can be generated. This allows your content sources to “fan out” to multiple logical asset types, each with its own distinct output configuration:

                             +------------------+       +------------------+
                             |                  |       |                  |
                             | Generated Assets +-----> |  Output Content  |
                             |                  |       |                  |
                             +---------^--------+       +------------------+
                                       |
+-------------------+        +------------------+       +------------------+
|                   |        |                  |       |                  |
|  Content Sources  +------> |  Content Assets  +-----> |  Output Content  |
|                   |        |                  |       |                  |
+-------------------+        +------------------+       +------------------+
                                       |
                             +---------v--------+       +------------------+
                             |                  |       |                  |
                             | Generated Assets +-----> |  Output Content  |
                             |                  |       |                  |
                             +------------------+       +------------------+

Output Configurations¶

Similarly to the “fan out” possible between content sources and the asset types generated from them; multiple output configurations can target the same asset types. It should start to be apparent the flexibility of Blog’s ability to generate static content:

                                                        +------------------+
                                                        |                  |
                                                    +---> Category Index   |
                             +------------------+   |   |                  |
                             |                  |   |   +------------------+
                             | Category Assets  +---+
                             |                  |   |   +------------------+
                             +---------^--------+   |   |                  |
                                       |            +---> Category Details |
                                       |                |                  |
                                       |                +------------------+
                                       |
                                       |                +------------------+
                                       |                |                  |
                                       |            +---> Article Index    |
+-------------------+        +------------------+   |   |                  |
|                   |        |                  |   |   +------------------+
| Markdown Sources  +------> |  Article Assets  +---+
|                   |        |                  |   |   +------------------+
+-------------------+        +------------------+   |   |                  |
                                       |            +---> Article Details  |
                                       |                |                  |
                                       |                +------------------+
                                       |
                                       |                +------------------+
                                       |                |                  |
                                       |            +---> Tag Index        |
                             +---------v--------+   |   |                  |
                             |                  |   |   +------------------+
                             |    Tag Assets    +---+
                             |                  |   |   +------------------+
                             +------------------+   |   |                  |
                                                    +---> Tag Details      |
                                                        |                  |
                                                        +------------------+

Base Context¶

The reading phase starts with the creation of a base context dictionary. This dictionary contains static information that may be useful to inside the templates of template oriented writers. Another possibility is of pipeline components basing their behavior on top-level values in the context. This is generally avoided when possible, favoring direct configuration of pipeline components, but some components do use top-level context variables as a convience in configuring many of them in a single place.

No keys in of the base context that share a name with any of your content types should be used, since this is where the assets of those content types will be stored.

Content Types¶

Before any part of the reading phase can begin a definition of your content types must be provided. This definition is provided as a dictionary of dictionaries. Each top-level key maps the name of a content type to its definition. Each type definition has three required keys:

• loader discovers the paths of candidate content sources
• reader parses the content of discovered sources into assets
• processors a list of asset processors to apply in order

A simple example for loading some Markdown articles might look something like:

content_types = {
  'articles': {
    'loader': blot.loaders.BasicLoader('content/articles', ['md]),
    'reeader': blot.readers.MarkdownReader(),
    'processors': [
      blot.assets.PathMetadata(),
      blot.assets.AutoSummary(),
      blot.assets.Humanizer('date'),
    ]
  }
}

Producing the Write Context¶

Once a base context and content types have been defined the reading process can be performed with blot.read():

write_context = blot.read(base_context, content_types)

---
title: Awesome Post!
---

Reasons to read this post:
  - it is awesome!
  - because!

Targeting¶

Each content asset as a non-metadata attribute target which is for storing the location on disk where the asset should be written to disk. Since Blot a static site generator, this location determines its URL as well. During the writing phase, all writers perform targeting on all of their assets before any writer performs rendering. This is to ensure that during rendering, the URLs of all asset types are available. Otherwise, only the URLs of assets already processed could be properly rendered.

If a writer will be rendering output for each targeted asset the target path pattern will generally contain one or more interpolation variables that name metadata properties. This allows the output path for each asset to vary uniquely so they don’t overwrite each other.

Rendering¶

Now that all assets have been properly targeted each writer undergoes rendering wherein it may process the content of assets for the last time. This will typicall involve combining the asset, with the global context and some specified template to produced a rendering of the asset. However, this isn’t nessecary writers are free to do whatever or nothing at all. At this point the writer shouldn’t be modifying assets or the global context at all.

Once the final content of an asset has been derived it is ready for writing to its target destination.

Performing the Writes¶

With the rendering context in hand blot.write() can be used to actually generate your site:

posts = context['posts']['assets']
categories = context['posts']['categories']
staticfiles = context['staticfiles']['assets']

writers = [
  blot.writers.ListWriter(posts, 'post.html', 'posts/{slug}/index.html'),
  blot.writers.IndexWriter(posts, 'posts.html', 'posts/index.html'),
  blot.writers.ListWriter(categories, 'category.html', '/{name}.html'),
  blot.writers.IndexWriter(categories, 'categories.html', 'index.html'),
  blot.writers.StaticWriter(staticfiles, 'static/{filename}'),
]

blog.write(context, writers)

Subpackages¶

Submodules¶

Getting Setup¶

Firstly, you’ll need to install Blot.

One that’s done, we recommend installing Fabric which will be useful for writing your site’s build script. You can read more about Fabric at their official overview but essentially it is a Python library that helps you write scripts that perform local or remote operations. It then exposes those operations with a nifty cli tool. Brace yourself for the shortest Fabric tutorial in the world:

Step 1: Write a Python file called fabfile.py and put functions in it:

# fabfile.py
def hello(name):
    print "Hello, {}!".format(name)

Step 2: Use the fab cli tool to invoke individual functions from the fabfile:

$ fab -l
Available commands:

hello

$ fab hello:world
Hello, world!

Done.

Listing the Goals¶

For this tutorial we want to build just enough functionality into the blog site to showcase how Blot is used. Let’s stick some typical blog features:

Chronological Posts

• Each post is a markdown file in a posts folder.
• There should be an index of the posts sorted chronologically
• There should be a page for each post where we can read it

Post Categories

• Each post has an assigned category
• There should be an index of the available categories
• There should be a detail page for each category listing the posts it contains

Static Assets

• There should be a logo image which we’re able to integrate into our theme.
• There should be a stylesheet which styles our pages

Getting Started¶

Create a new directory for your site and within it create a fabfile.py file, a posts directory and a theme directory:

|-- fabfile.py
|-- posts/
`-- theme/

In the theme directory, create the file style.css. This will be our first content source. The goal with static files is simple: copy the file from the source to the destination.

Open your fabfile.py and enter the following contents:

base_context = {}

The base_context is like the starting-point for data in the build process. Anything you add here will be subject to modification by anything in the pipeline. Settings placed here can also affect the behavior of components in the pipeline. For now, we’ll leave it empty.

Content Types¶

Add the following to fabfile.py:

content_types = {
    'stylesheets': {
        'loader': blot.loaders.BasicLoader('theme', extensions=['css']),
        'reader': blot.readers.StaticReader(),
        'processors': [
            blot.assets.PathMetadata(),
        ],
    }
}

The content_types dictionary holds all of the content type definitions for our site. Currently, there is only a single type stylesheets. Let’s take a look at each of its keys:

Reading¶

Now that we have a base context and our content type definitions, reading can be performed with blog.read(). This will load our sole stylesheet, add some metadata to it. Finally, an updated version of the base context will be returned. Add the following to fabfile.py:

def build():
   context = blot.read(base_context, content_types)

We now have a function in the fabfile.py that we can invoke from the commandline. However if we do, we wont see anything meaningful just by loading content. Let’s add a quick debugging print to see the results of the reading process:

def build():
   context = blot.read(base_context, content_types)
   print context

Now we can invoke the build function with the fab command-line utility:

fab build
{'stylesheets': {'assets': [<blot.assets.base.ContentAsset object at 0x7fc3d9041c50>]}}

Done.

We can see that the context now contains a stylesheets key that maps to the context for that content type. Within that context is an assets key that contains the actual processed content assets.

If we change the print statement to print context['stylesheets']['assets'][0].metadata we can get a closer look at the asset itself and evidence of blot.assets.PathMetadata processor at work:

fab build
{'ancestry': '', 'parent': 'theme', 'extension': '.css', 'basename': 'style.css', 'dirname': 'theme', 'filename': 'style'}

Done.

For more information on loaders, readers and asset processors visit the Reading Content section of the User Guide.

Writing¶

Writing involves taking some assets and passing them to a writer. In our case we don’t need to render anything so we can use the blot.writers.StaticWriter to simply write our stylesheet asset contents to disk:

def build():
    context = blot.read(base_context, content_types)

    stylesheets = context['stylesheets']['assets']
    blot.write(context, [
        blot.writers.StaticWriter(stylesheets, 'static/{basename}'),
    ])

First, we grab the stylesheet assets out of the context. Then we call blot.write() while passing it the context and a list of writers. In this case, the only writer involved.

blot.writers.StaticWriter takes two arguments:

• The assets to write
• A path pattern describing destinations

As StaticWriter goes to write each asset it will determine where to write it by interpolating the asset’s metadata into the provided path pattern. The path pattern we’re using static/{basename} requires that the assets being written contain this metadata property. Luckily, basename, among others, is provided by the PathMetadata asset processor we used during reading. For our style.css file, we should expect its destination path to be static/style.css.

If we invoke the build function from the commandline that is exactly what we should see in the newly created output directory:

$ fab build

Done.

$ tree output
output
`-- static
    `-- style.css

1 directory, 1 file

For more information on writing visit the Writing Assets section of the User Guide.

Intermission¶

By now you should be developing a clearer idea of how you can bring together loaders, readers, processors and writers to form a pipeline for your content sources with Blot. The process for handling the rest of our content types works just like it does for our stylesheets. Really!

At this point the tutorial will start moving a lot faster.

Before moving on, let’s add small helper function to fabfile.py to clean our output directory by deleting it. The whole script should appear as follows:

from fabric.api import local

import blot

base_context = {}

content_types = {
    'stylesheets': {
        'loader': blot.loaders.BasicLoader('theme', extensions=['css']),
        'reader': blot.readers.StaticReader(),
        'processors': [
            blot.assets.PathMetadata(),
        ],
    }
}


def clean():
    local("rm -r output")

def build():
    clean()
    context = blot.read(base_context, content_types)
    stylesheets = context['stylesheets']['assets']
    blot.write(context, [
        blot.writers.StaticWriter(stylesheets, 'static/{basename}'),
    ])

We’ve added a clean function which removes the output directory which allows us to invoke it from the commandline. It uses a function from the Fabric api, local, which we’ve imported at the top. It makes running local shell commands very convienent. As a final note, we’re calling clean first thing from build which gives a clean target path each time you build.

Introducing Posts¶

To introduce our actual blog posts, we’ll add a new posts content type:

content_types = {
    'stylesheets': {
        'loader': blot.loaders.BasicLoader('theme', extensions=['css']),
        'reader': blot.readers.StaticReader(),
        'processors': [
            blot.assets.PathMetadata(),
        ],
    },
    'posts': {
        'loader': blot.loaders.BasicLoader('posts', extensions=['md']),
        'reader': blot.readers.MarkdownReader(),
        'processors': [
            blot.assets.PathMetadata(),
        ],
    }
}

This is very similar to the stylesheets content type definition, but with a few changes. First, we want to load files from the posts directory this time and only files with a .md file extension. Any such files will then be parsed as markdown by the blot.readers.MarkdownReader.