Ideal Ractive workflow

[MartinKolarik]

So, I started using gobble more (btw, it's really great!), and discovered a couple of things I'd like to discuss.

Because of Ractive components, I use inline scripts and styles very often. Since I wanted to be able to use ES6, I wrote a small script which extracts content of <script> elements from html files, transforms it with babel, and writes ES5 code back to the html file.

I wanted to submit a PR to babel/gobble-babel, but then I realized we'd need to know the file name (to determine whether we're dealing with a js or html file), and an ability to set the file extension from within the transformer. Is seems the file name is available as this.src, but I haven't found a way to change the extension.

The question is if it wouldn't be better to have another gobble plugin to deal with inline scripts, but that would effectively mean writing all js and css transformers and checkers (you might want to lint your inline scripts too!) twice. I think that would be a hell to maintain for both plugin authors and consumers, not to mention that two different plugins doing almost the same thing could have different APIs.

Since file transformers can already return an object with the code and a source map, I was thinking if adding ext to this object to change the file extension for the current file would make sense.

Also, while writing plugins for jshint and jscs, I was wondering if having file observers wouldn't be useful to prevent linting files that didn't change.

[Rich-Harris]

My own build definitions for static file projects typically look something like this:

module.exports = gobble([
  // src/app contains both .js and .html files
  gobble( 'src/app' )

    // first, convert all .html to .js
    .transform( 'ractive', { type: 'es6' }) 

    // *then* transpile with babel - I use .babelrc for options
    // - e.g. 'es6.modules' is blacklisted
    .transform( 'babel' )

    // then make a bundle of all the ES6 app code, including components.
    // allowing babel to convert to CommonJS and running the whole lot
    // straight through Browserify is also a viable option
    .transform( 'rollup', {
      entry: 'main.js',
      dest: 'bundle.js',
      external: [ 'ractive', 'some-other-module' ],
      format: 'cjs',
      sourceMap: true
    })

    // unfortunately this is necessary at the moment, otherwise
    // Browserify gets dreadfully confused. I plan to fix it in
    // rcu-builders
    .transform( 'derequire' )

    // then Browserify creates the final bundle, including all npm
    // dependencies including ractive
    .transform( 'browserify', {
      entries: [ './bundle' ],
      dest: 'bundle.js',
      standalone: 'myApp',
      debug: true
    }),

  gobble( 'someOtherStuff' )
]);

It's a little bit convoluted (par for the course for web development circa 2015...) but it gets the job done - no need to extract the <script> portion of a template. And if everything is working correctly, you get sourcemaps right back to the .html files. It doesn't give you the opportunity to lint or transform (e.g. autoprefix) your CSS - maybe gobble-ractive should support that somehow.

an ability to set the file extension from within the transformer. Is seems the file name is available as this.src, but I haven't found a way to change the extension.

Poorly documented, but this can be done by passing through accept and ext options - some file transformer plugins have default options for this, e.g. gobble-ractive, but you can do it in the build definition:

node.transform( someTransformer, {
  accept: '.in-ext',
  ext: '.out-ext'
});

Returning an ext option with code and map is quite a nice idea though. Hmm...

Also, while writing plugins for jshint and jscs, I was wondering if having file observers wouldn't be useful to prevent linting files that didn't change.

Yes! Would definitely like to revisit this whole area - the pieces are basically all there, they just need putting together. It's something that would make my own life a lot easier so it'll definitely happen eventually :-)

[MartinKolarik]

// first, convert all .html to .js
.transform( 'ractive', { type: 'es6' }) 

// *then* transpile with babel - I use .babelrc for options
// - e.g. 'es6.modules' is blacklisted
.transform( 'babel' )

I was considering this as well, but it doesn't allow you to lint the code (at least style check) and use imports inside your scripts, since these have to be at the top level and the scripts generated by gobble-ractive are wrapped in IIFE (although I could live with using require there). On the other hand, transforming the JS code inside HTML would break sourcemaps, that's something I haven't thought of (although, again, these could probably be easily fixed)...

Poorly documented, but this can be done by passing through accept and ext options - some file transformer plugins have default options for this, e.g. gobble-ractive, but you can do it in the build definition

I know about this but if the transformer could handle both .html and .js files, you'd want to keep .html for html files and .js for js files.

[MartinKolarik]

Oh yes, there was one more reason why I wanted to run babel before gobble-ractive. I'm using type="text/ecmascript-6" for the inline scripts, which is required to get support for ES6 syntax in WebStorm, so rcu doesn't recognize them. But maybe that's something we could fix in rcu?

[Rich-Harris]

which is required to get support for ES6 syntax in WebStorm, so rcu doesn't recognize them

Ah, gotcha. Sounds like a bug in WebStorm to me! But yeah, that could be fixed in rcu. A generic approach would probably consist of passing in an option like this...

{
  transpile: {
    'ecmascript-6': transformWithBabel,
    coffeescript: transformWithCoffeeScript
  }
}

...and so on. A couple of Browserify transforms already support this sort of thing, would make sense to have it supported at an rcu level and be totally pluggable.

I was considering this as well, but it doesn't allow you to lint the code (at least style check) and use imports inside your scripts

Yeah, I guess rcu needs a pluggable linter as well. The import thing is a source of pain to me also - since drinking the ES6 Koolaid I really like named imports and exports since you can generate very efficient bundles with them, and require() is a fly in the ointment. Ideally we'd phase require() out in favour of import, but it's problematic in the ractive-load case, since it would be illegal JavaScript. In node.js it's not a problem, since it's a fairly trivial AST transformation, but in the browser you probably don't want to include the whole of acorn (or whatever). Not entirely sure what the best solution is.

I know about this but if the transformer could handle both .html and .js files, you'd want to keep .html for html files and .js for js files.

Doesn't this work if you just don't pass in the accept and ext options?

[martypdx]

I moved away from the single file format because there's more flexibility in using split windows in the IDE versus the scroll up and down dance I found myself doing as the components became more complex.

Sticking to single files also makes it trivial to do preprocessing on the files:

var components = gobble( 'assets/components' )
    .transform( sass, { includePaths: sassIncludes } )
    .transform( 'babel' )
    .transform( makeComponent )
    .transform( 'ractive', { type: 'cjs' } );

Here's the makeComponent gobble task I use that rolls them up. It allows nesting in folders for easier organization (we're up to 40+ components). It's from the broccoli days, so the file access is still sync, someday sander I promise :)

var fs = require( 'fs' ),
    path = require( 'path' ),
    join = path.join;

module.exports = function makeComponent ( src, outputdir, options, callback ) {

    function isDirectory( dir, f ){
        return fs.lstatSync( join( dir, f ) ).isDirectory();
    }

    function readFile( dir, name, ext ){
        var p = join( dir, name + '.' + ext );
        return  fs.existsSync(p) ? fs.readFileSync( p, 'utf8' ) : '';
    }

    function wrapFile( tag, file ){
        return file ? '\n\n<' + tag + '>\n' + file + '\n\n</'+ tag + '>\n' : '';
    }

    function writeComponent ( src, c, list) {
        var dir = join( src, c );
        var file = readFile( dir, c, 'html' );
        file += wrapFile( 'style', readFile( dir, c, 'css' ) );
        file += wrapFile( 'script', readFile( dir, c, 'js' ) );
        if ( file ) {
            var write = join( outputdir, c + '.html' );
            fs.writeFileSync( write, file );
            list.push( c );
        }
    }

    function writeDirectory ( dir, list ) {
        var cdir = fs.readdirSync(dir);

        var components = cdir.filter(function(each){
            return isDirectory( dir, each );
        });

        var childpath;

        components.forEach(function(c){
            writeComponent( dir, c, list);
            childpath = join( dir, c );
            writeDirectory( childpath, list );
        });
    }

    try {
        var list = [];

        writeDirectory( src, list );

        var listFile = "var Ractive = require( 'ractive' );\n\n";
        list.forEach( function(c) {
            listFile += "Ractive.components['" + c + "'] = require( './" + c + "' );\n";
        });

        fs.writeFileSync( join ( outputdir, 'components.js' ), listFile);

        callback();
    } catch(err){
        callback(err);
    }
};

I also create a components.js file that globally registers components and I don't have to think about anything more than adding the files to the project.

I'm currently using browserify for both my js and external modules - but doing so explicitly:

var excludeModules = [ 'moment', 'ractive', 'firebase', 'superagent' ];

var bundle = gobble( [ components, js, vendor ] )
    .transform( 'browserify', {
        entries: './index.js',
        configure: function ( bundle ) {
            excludeModules.forEach( function( module ) {
                bundle.exclude( module );
            });
        },
        dest: 'bundle.js',
        debug: gobble.env() !== 'production'
    });

var modules = gobble( 'assets/js/passthru' )
    .transform( bundleModules, { modules: excludeModules });
        // bundleModules is just browserify.require

My next step is to replace the first use of browserify with esperanto-bundle modified to output the imports list - which I'll then run the browserify require bundle on.

[MartinKolarik]

@Rich-Harris

Sounds like a bug in WebStorm to me!

Yeah, I told them it should work with the default type, but it seems we'll have to wait for that.

The import thing is a source of pain to me also - since drinking the ES6 Koolaid I really like named imports and exports since you can generate very efficient bundles with them, and require() is a fly in the ointment. Ideally we'd phase require() out in favour of import, but it's problematic in the ractive-load case, since it would be illegal JavaScript. In node.js it's not a problem, since it's a fairly trivial AST transformation, but in the browser you probably don't want to include the whole of acorn (or whatever). Not entirely sure what the best solution is.

I'm not sure about ractive-load, but for bundling with esperanto, it should be enough if ES6 rcu-builder wasn't wrapping the code in IIFE.

Doesn't this work if you just don't pass in the accept and ext options?

Ha! Didn't even occur to me to try that.

@martypdx

Hmm, this looks like a good idea. That scrolling thing really is an issue with some components, and having separate files means no issues with transforming or linting the code. You won't get source maps to original ES6 code, but babel's output isn't that horrible, and I think it could be solved by using rcu directly instead of gobble-ractive. Will definitely give this a try.

[martypdx]

You won't get source maps to original ES6 code ... and I think it could be solved by using rcu directly instead of gobble-ractive

I'm behind on understanding how the source maps propagate, is it that gobble-ractive isn't "looking" for upstream sourcemaps? Let me know if you figure it out 😁

but babel's output isn't that horrible

yeah, it's not too bad. key thing is you get Cmd+P support in chrome to load the component js file.

having separate files means no issues with transforming or linting the code.

It makes it real easy to do things like append a standard @import '_ourstuff'; ( or whatever flavor of css you use ) and otherwise reduce boilerplate.

[Rich-Harris]

it should be enough if ES6 rcu-builder wasn't wrapping the code in IIFE.

true - might be easier than I thought. In fact, maybe ractive-load support is as simple as doing a regex-based replacement of import statements with the equivalent require statements, and hoping that people don't comment them out or have strings that get caught by the regex. (That's horrible, I know, but we're already doing something similar to discover require calls in the first place for AMD modules...)

is it that gobble-ractive isn't "looking" for upstream sourcemaps?

Not exactly - the theory (and it is a theory - most of the time it works, but sourcemaps really can be quite fantastically difficult to debug when they go wrong) is that each transpilation step only cares about generating its own sourcemap, and gobble stitches them together (with sorcery) at the end. If a step doesn't generate a sourcemap, sorcery considers its output to be an original source. At present, sourcemaps are only generated with ES6 output - haven't got round to enabling it for other formats (though the logic is basically sitting there waiting to be activated).

I can definitely see the appeal to having components stitched together from multiple files. Most of my work is done on a laptop, where I don't have room for multiple panes, so I tend to rely on code folding instead which is why I haven't found it as big a source of frustration. But the linting/processing/transpilation etc is a solid reason to do it, and I don't see any good reason not to support it at the rcu level - maybe something like this...

<link rel='ractive' href='./OtherComponent.html'>

<p>some markup</p>

<style>
  @import './typography.css'; /* shared files = better modularity/composition! woo! */
  @import './buttons.css';
  p { color: purple }
</style>

<script src='./widget.js'></script>

...maybe with hooks for postCSS etc (or maybe they're no longer necessary)

[MartinKolarik]

I'm behind on understanding how the source maps propagate, is it that gobble-ractive isn't "looking" for upstream sourcemaps? Let me know if you figure it out

The problem is that gobble-ractive is consuming the HTML files you generated with makeComponent, and there are no source maps for these files (I don't even think you can have source maps for HTML files).

and hoping that people don't comment them out or have strings that get caught by the regex

Actually, it isn't that hard to prevent this. There are, I think, four things that could cause problems: strings, comments, RegExps, and ES6 shorthand method names. Filtering any of these separately without an AST is tricky (you can have something that looks like a comment inside a string, etc.), but we need to ignore all of them, and that's easily achievable with regular expressions.

I don't see any good reason not to support it at the rcu level

👍