06-03-2015
At Demandware, I work on a reference application – sort of like a starting point for developers to use in building a e-commerce web application. Its purpose is to cut down on implementation time. One of the demands for this project is to create something that can be used and liked many developers with varied backgrounds, experiences and opinions.
When I decided to introduce a much needed modern build tool into the project (for things like compiling Sass to CSS, bundling browserify modules, running tests etc.), I was met with a choice of whether to use grunt or gulp.
While grunt has grown to be quite popular among many developers (the ones at my company included), there are limits to its approach that make it clunky to work with (a declarative API in defining tasks, different tasks interact with each other through the file system). Gulp has gained quite a bit of popularity due to its approach of using stream and a programmatic API. Admittedly, gulp is a lot more fun for me to use and maintain.
However, this was not a straightforward choice as many people were voicing concern against going after the “latest and greatest” tool, and wanting to keep the technology stack consistent within the enterprise. While this is not an argument I agree with, I could appreciate and respect the intention.
Instead of plunging into a heated debate with other engineers over what is a better build tool, I decided to support both for the project. In retrospect, this decision has turned out to be a wise one.
It makes the idea of introducing a build tool that much easier to sell. I have the ability of pitch it by saying something like “you can use either gulp or grunt, and they both would work the same”. It appeals to three groups – the ones who have spent time and effort learning grunt and feel comfortable in it, the ones who keep more up to date with bleeding edge, and also the ones who do not know either but want to learn/ use one.
Furthermore, it sets the expectation that while a build tool is needed and important in modern web development, the project is not tied to a single tool. This opens up room for experimentation with new workflow and tooling options. If a new tool is available, it should be adopted and played with, regardless of whether it is a grunt or gulp tool. This also means that a completely new build tool can be experimented with as well, be it broccoli, npm run-scripts, or (gasp!) even make. We promise that the interface and tasks should be eventually consistent across these build tools, but they do not have to be on parity at any single time. We do try however to make the most basic tasks such as Sass and browserify to be available across all build tools.
While this setup makes a lot of sense for our project, it might not be as practical or wise for others, especially those that are used/ maintained by a small group of people whose preferences/ opinions are more closely aligned. Having said that, I find it quite liberating and welcoming to newcomers to a project to have different build tools available. It is sort of like a quiet nod to all the three groups I mentioned above, making sure that none of them feels particularly left out. I have come to strongly believe that in order to make a project successful, it is usually helpful to make contributors, especially newcomers, feel welcomed and comfortable.
11-17-2014
About a year and a half ago, amidst the hype of static sites, I attempted to convert my WordPress blog over to a static blog. The advantages were clear to me as a developer: much faster performance, authoring in markdown, source-controlled content out of the box, easy and free hosting with Github Pages, and the list goes on. This was the motivation for tobiko, a Grunt-based static site generator – a tool I wrote that I thought would be useful for other developers out there.
Existing static site generators
The path to tobiko where it is today was not always straight forward however. I did not want to reinvent the wheels initially, and tried out a few different static site generators out there.
I began with the obvious choice – Octopress powered by Jekyll. There was (and still is) a lot of enthusiasm about Jekyll in the open source community, and I believe that is so for good reasons. However, at this time period, I started to write a lot more JavaScript in the browser as well as node, and for some reason Ruby and its tool chain never really clicked for me. I had a lot of issues setting Octopress up and deploying properly, most likely due to my own shortcoming, despite having taken an intro to Ruby course on Codeacademy a few months prior.
As node was already a huge and growing community back then, there were a few contenders of static site generator written in node as well. Most notably was wintersmith, which I thought was very cool. I tried it out and even wrote a plugin for it. Nonetheless, I ended up parting ways with wintersmith as well. The tool was authored mostly in Coffeescript, which did not appeal to me, and the whole plugin architecture seemed confusing.
Grunt – a proven build tool
At the same time, Grunt was quickly becoming very popular as a mainstream build tool with a great community around it. And I thought the whole idea of a static site generator fits in very well with a build tool ecosystem. So I created tobiko – a Grunt-based static site generator.
Main features
I wanted tobiko to be based on a stack that I would like to work with. Here are a few highlights:
- tobiko itself is a glorified grunt plugin, which features 2 main task – parsing content and generating html pages.
- Content can be written in either Markdown (with YAML frontmatter) or JSON.
- Templates are written in Handlebars by default.
- Styles are written in SCSS by default.
- JavaScript uses RequireJS to support AMD syntax.
- Site and project configurations are stored in JSON files that can be overwritten based on the environment (
dev vs prod).
- Content can be nested indefinitely in the traditional file system. HTML pages reflect that nesting structure. That path however can be overwritten as desired.
- Pagination and archives.
- Image compression with responsive image support by generating multiple image resolutions on build.
- Deployment to Github Pages or any static server with
rsync.
Yeoman generator
If these features are enough to intrigue you, I’d encourage you to try tobiko out. I’ve also written a Yeoman generator for tobiko, so that you can set up all the directory structure and grunt tasks within a minute or two. The whole idea is to scaffold and deploy a simple static site within a matter of minutes.
Roadmap
I have used tobiko to create a few sites so far with very satisfying results. I think that something like tobiko can be very useful for a simple business site, personal blog or even a style guide for larger projects. It is definitely meant for developers who know their way around code and JavaScript, and who enjoy writing content in Markdown.
In the course of using tobiko, I have found a few areas for improvement as well as possible ways to expand the project. Here are some ideas of what I’d like to do with it next (without selling too much vaporware):
- Create a gulp-based version. File systems are great, but streams can speed up the build process many times over.
- Add more tests to make the tasks more stable.
- Support for browserify/ CommonJS syntax for JavaScript.
- Support for other templating engines.
If you take a look at tobiko (which is by itself pretty simple) and think you can help out with these items, please feel free to file an issue or send a pull request.
Thanks very much for reading this far, and if I’ve got you somewhat convinced, definitely give tobiko and generator-tobiko a try!
09-20-2014
As I build more web applications, one problem that I keep running into is how to include application configurations into the application source code.
What do I mean by configurations?
They include application-specific settings such as the application name, API URL, debug mode etc. These are metadata to the application, similar to the metadata contained in package.json for a Node module.
As I develop most of my applications locally, some of these settings will differ when the application is served on localhost versus when served in production. For eg., I usualy use a staging API server for local development and then a production server for deployment.
I could certainly hardcode these settings into the source code right?
The fact that I need 2 (or more) environment-specific versions of these settings make them difficult to be hardcoded. Furthermore, as a developer, I feel a little “dirty” when I hardcode these settings. Ideally, they should be declared in a JSON file somewhere, just like package.json.
There must be other solutions out there!
Surprisingly, there isn’t a good straight forward solution out there, as far as I know (if you know of one, please leave a comment). Here are a few worth mentioning:
One can follow the approach outlined by Addy Osmani in making environment-specific builds with build tools and use some sort of string replacement plugin. However, this will introduce some ‘not-so-pretty’ delimiters in the codebase, such as @@foo used by grunt-replace.
Henrik Joreteg’s clientconfig is another elegant solution and very close to what I have in mind, however it relies on a server component to set all the configurations in a cookie. This is not a hard requirement, but is not feasible for a static site running on a simple file server (such as Github Pages) like this blog.
Another attempt at solving this problem
If you are using browserify to bundle your JavaScript code, this might be a good solution for you.
Using aliasify, you can require your config, which is declared in a config.json file like this:
{
"API_URL": "http://example_server.com/api",
"env": "development"
}
var config = require('config');
console.log(config.API_URL); //prints out API_URL
if (config.env === 'development') {
// do something specifically in development mode
}
To declare this transformation, you can add the following section to package.json:
{
"aliasify": {
aliases: {
"config": "./config.json"
}
}
}
Pretty simple, right?
Environment-specific configs
We can make this even better, by allowing for environment-sepcific configs. To do that, I recommend using a build tool like grunt or gulp. I am sure there is probably a way to do it with just the simple browserify command line interface, but using browserify programmatically makes things a lot easier.
Here’s an example of how I am using this in a recent project:
// gulpfile.js
// enable development mode
var dev = false;
gulp.task('enable-dev-mode', function () {
dev = true;
});
gulp.task('scripts', function () {
// set up browserify bundle in here
// see https://github.com/gulpjs/gulp/blob/master/docs/recipes/browserify-uglify-sourcemap.md
// aliasify config
var aliasify = require('aliasify').configure({
aliases: {
'config': './config' + (dev ? '.dev' : '') + '.json'
},
configDir: __dirname
})
bundler.transform(aliasify);
});
In this approach, aliasify will automatically use config.dev.json for development environment and config.json for production one.
A similar approach could be taken when using grunt-browserify. In fact, it is probably easier to do environment-specific stuff with grunt due to its ability to declare a task target.
There you have it, a simple approach to accessing environment configuations in your project that is made possible by browserify.
What’s bad, what’s good, what could be better?
This solution’s downside is relying on browserify (and possibly a build system), which might not be applicable for an older project. If you use RequireJS/AMD, I imagine it’d be similarly easy to include such a config file using the JSON plugin.
One possible improvement to this solution is allowing the ability to cascade configs across environment. For example:
In config.json
{
"appName": "FooBar",
"apiUrl": "http://foobar.com/api"
}
And then config.dev.json just “extend” it without redeclaring common configs:
{
"apiUrl": "http://localhost:3000"
}
This is currently not possible with using aliasify. I have implemented a similar version of this as a grunt task for tobiko, but it’s not polished enough to be used by others yet. I will update this space if and when this becomes available.
03-31-2014
Integrating Handlebars templates in modern web development workflow has been crucial in helping me organize my code neatly and maintain sanity. I want to achieve a few things:
- avoid the
<script> tags to take advantage of syntax highlighting,
- break large templates into smaller files that can be managed and reused (componentized),
- automatically compiles and ready for use, and
- namespaced with file names as template names.
In a current Backbone.js application at work, which is set up with grunt and RequireJS (you can find a similar set up with the default yeoman backbone generator), I have found the following set up to meet most of these goals.
I can use the templates with RequireJS in the views like so:
// viewOne.js
// I use commonjs syntax with require
// http://requirejs.org/docs/whyamd.html#sugar
var Templates = require('templates');
var viewTemplate = Templates.viewOne;
var html = viewTemplate({data: 'test'});
The template directories look something like this
app/templates/
├── viewOne.hbs
├── viewTwo.hbs
├── viewThree.hbs
├── ...
└── viewTwentyThree.hbs
0 directories, 23 files
I use grunt-contrib-handlebars to compile the templates. Here’s the Gruntfile.js config for local development:
handlebars: {
compile: {
options: {
amd: true,
namespace: 'Templates',
partialsUseNamespace: true,
processName: function(filePath) {
var file = filePath.replace(/.*/(w+).hbs/, '$1');
return file;
}
},
files:{
'.tmp/scripts/templates.js': ['<%= yeoman.app %>/templates/*.hbs']
}
}
}
The above code does a few things. More detailed explanation of what each options does can be found on the plugin github page.
- The
processName function takes the files and use their name as the template name.
- With
partialsUseNamespace turned on, you can use partials both within your template as > partial or through the global Templates object.
In the build step for deployment, instead of just outputing the compiled templates into the .tmp folder, RequireJS can put Templates in the optimized javascript with the following Grunt task config (note the paths option):
requirejs: {
dist: {
options: {
baseUrl: '<%= yeoman.app %>/scripts',
optimize: 'none',
paths: {
'templates': '../../.tmp/scripts/templates'
}
}
}
}
This simple set up has allowed me to rapidly develop a complex apps by breaking down complex templates into small reusuable components and use them liberally anywhere with RequireJS, yet keep the built JavaScript minimal and compressed. I hope it will be useful for others who might be looking for a similar solution. If you find this useful or would like to suggest an improvement, please leave a comment.
