nunjucks for Fun and Profit

I am using a lot of code generation at work. I started this stuff back in 2004, using good old Python. Admittedly a bit late in the game I discovered jinja2, and it was great. I converted a couple of things, but then I rewrote everything using nodejs, and it was actually a ton of fun - and much faster than Python, too. Now, Python and I go way back to the early 2000s, but really we've parted ways and these days Javascript looks more sexy than ever.

Long story short, I've written my very first ever npm module: nunjucks-grime.

So nunjucks-grime: what is it?

Some nunjucks extensions for code generators

Granted, that sounds kind of underwhelming. So what is it?

Stitching sources

The way you normally use nunjucks is this:

template ---> generated output

This incredible piece of ASCII art means that the default mode of operation is that the template must be a complete description of the output file, where generated content is added in.

But sometimes you are in a situation where you need this:

(existing file + something generated) --> generated output

For example, in my code base I have existing C/C++ code that includes stuff dynamically generated and stuff manually maintained. So I need a solution that

This is not a solution nunjucks supports out of the box, so I added a small syntax extension. The elements of the solution are these:

Stitching Example

Ok, let's assume you have this file stitched.h:

The text you want to modify

#ifndef STITCH_SOURCE_H
#define STITCH_SOURCE_H

... tons of code here that is NOT generated ...

// BEGIN GENERATED FUNCTIONS: DO NOT MANUALLY EDIT, THIS IS GENERATED CODE
// BUT how do I get my generated stuff in here while retaining all the existing bits? 
// END GENERATED FUNCTIONS: DO NOT MANUALLY EDIT, THIS IS GENERATED CODE

... huge megatons of code here that is NOT generated either ...

#endif // STITCH_SOURCE_H

Fear no more, nunjucks-grime is here. You define a special stitch template like this:

The stitch-template dialect

{% stitch %}
// BEGIN GENERATED FUNCTIONS: DO NOT MANUALLY EDIT, THIS IS GENERATED CODE

{%- for item in well_known_client_functions %}
typedef HRESULT (*WINAPI LPFN{{ item }})(LPCTSTR Bassdrum);
{%- endfor %}        

// END GENERATED FUNCTIONS: DO NOT MANUALLY EDIT, THIS IS GENERATED CODE
{% endstitch %}

Some points to notice:

The actual code

If you are familiar with nunjucks (or jinja2 for that matter), this should be pretty readable. Next up: putting it all together

var nunjucks = require('nunjucks-grime');

var env = new nunjucks.Environment(new nunjucks.FileSystemLoader('views'));
env.opts["stitch_source_loaders"] = [new nunjucks.FileSystemLoader('stitch_sources')];
console.log(env.render('stitched.h', data));

This will modify your existing source like this:

#ifndef STITCH_SOURCE_H
#define STITCH_SOURCE_H

... tons of code here that is NOT generated ...

// BEGIN GENERATED FUNCTIONS: DO NOT MANUALLY EDIT, THIS IS GENERATED CODE

typedef HRESULT (*WINAPI LPFNfoo)(LPCTSTR Bassdrum);
typedef HRESULT (*WINAPI LPFNbar)(LPCTSTR Bassdrum);
typedef HRESULT (*WINAPI LPFNblubber)(LPCTSTR Bassdrum);

// END GENERATED FUNCTIONS: DO NOT MANUALLY EDIT, THIS IS GENERATED CODE

... huge megatons of code here that is NOT generated either ...

#endif // STITCH_SOURCE_H

That means you can run that repeatedly, as long as you want: nunjucks-grime will only ever touch the bits enclosed in BEGIN GENERATED FUNCTIONS.. and END GENERATED FUNCTIONS. Not so bad, eh?

OK, now that you are convinced, let's slowly review the code lines I used: 

var nunjucks = require('nunjucks-grime');

Note that nunjucks-grime is designed t be used as a drop-in replacement for the original nunjucks.

var env = new nunjucks.Environment(new nunjucks.FileSystemLoader('views'));

That was standard. 

env.opts["stitch_source_loaders"] = [new nunjucks.FileSystemLoader('stitch_sources')];

Now this is an extension: you must define how nunjucks-grime should find the original sources to be stitched. You can use any of the existing nunjucks loaders, or write your own standard-conforming loader.

console.log(env.render('stitched.h', data));

Look ma, no new parameters

Minor stuff

Also, because I needed it I have added two helper filters that jinja2 has supported for quite some time:

This was probably just an oversight of the nunjucks port: because on the web you don't normally need that kind of thing.

Example

Let's say you have this input:

#ifndef NUNJUCKS_EXTRAS_H
#define NUNJUCKS_EXTRAS_H

// By default, the list will not be aligned:
{%- for item in list_of_items %}
#define {{ item.name }} {{ item.value }}
{%- endfor %}

// but now you can left-align
{%- for item in list_of_items %}
#define {{ item.name|ljust(20) }} {{ item.value }}
{%- endfor %}

// and right-align
{%- for item in list_of_items %}
#define {{ item.name|rjust(20) }} {{ item.value }}
{%- endfor %}

#endif // NUNJUCKS_EXTRAS_H

Note the use of ljust and rjust to specify an alignment. The actual result with nunjucks-grime this: 

#ifndef NUNJUCKS_EXTRAS_H
#define NUNJUCKS_EXTRAS_H

// By default, the list will not be aligned:
#define foo 0x10
#define bar 0x1000
#define blubber 0xFFFFFFFE

// but now you can left-align
#define foo                  0x10
#define bar                  0x1000
#define blubber              0xFFFFFFFE

// and right-align
#define                  foo 0x10
#define                  bar 0x1000
#define              blubber 0xFFFFFFFE

#endif // NUNJUCKS_EXTRAS_H

What is up with the name?

I've been hooked on listening to this on volumes that shouldn't be legal.