SPAC: Publishing Apps

SPAC is a custom JavaScript framework for client-side, single-page web applications. It stands for "Stateful Pages, Actions and Components". Its design goal is to provide robust and simple entities that help you to structure apps. Pages and components provide the HTML, JavaScript functions and UI interactions. Actions govern external API calls. You define these entities in plain JavaScript, load up the central controller, and your app is ready to be served. Read the development journey of SPAC in my series: https://admantium.com/category/spac-framework/.

The past articles explained the components and features of SPAC. Now it’s time to go through the steps of publishing your app.

Assumptions & Observations

At the beginning of the development journey, I envisioned the following:

SPAC.js is intended to be a lightweight, self-initializing framework for publishing client-side applications. Its foundation is the controller, the central entity that resolves pages, components and actions. The controller is the only tool that you need to include in a JS. When started it, will serve the index page.

Over the last weeks, each entity was developed and supported by an extensive test suite. Then I assembled a very simple demo app with one page, components and action. And... the framework just did not work.

The troubles I ran into were interesting and rewarding to understand and solve. This blog post is about these learnings.

Modules: (ES5 != Node)

The module syntax of node is not the same as that of ES5 or ES6 that runs in a browser. The expressions require() and module.exports are node specific. My code base was using Node exports, which were incompatible with ES5. I was unaware of this issue because the unit tests relied on Babel to transpile the code to a compatible version. So, to make it running in a web browser, I need to transpile to that environment. Which leads me to ...

Imports & Bundling

A bundler takes a complex project, computes a dependency graph, and provides an optimized, transpiled for the target environment compatible version of your project.

Webpack was the first bundler that came to my mind. I absorbed the documentation, provided a sample config, and could transpile my projects. Then I created a simple HTML page and manually included the transpiled version. This version could not be executed because the Node module export syntax could not be understood by native webpack. From a blog post I understood that a Babel config is only applied after Webpack bundled the code, so my original problem could not be solved.

After some more research, I discovered Snowpack - and it could transpile my code base without any additional configuration. All SPAC entities were available in the browser. And then I executed Controller.init() which uses the Node module fs to recursively traverse files in a directory. For the time being, I tried to get the fs node module working with snowpack, following this documentation about polyfilling NPM packages, but could not get it working.

Stop for a moment.

Javascript running in a browser should not be allowed to traverse local files. This traversal is server-side, not client side!

This finding is particularly interesting. Initially, I considered these options:

• Dynamic Imports: ES5 supports a dynamic import() statement. This statement needs a filename, and this file is asynchronously fetched from the server. Therefore, the server-side actually needs to deliver individual JavaScript pages.
• Pre-Build Imports: Before bundling the app, I use a helper script that traverses the app directories, determines the pages, and then adds them to an inventory file. During bundling, the controller reads the inventory, and executes static imports of these file. Then, the application is bundled.
• Static Imports: All entities, including pages, need to statically import their required assets. Then, bundling "just" packs the application into the target format.

After some thought, and another try to create a self-initializing controller, the solution became a combination of all the above ideas:

• Inventory: Before building, a script detects all pages, and creates a file called inventory.js
• Imports: During the init phase, the controller loads all required pages from the inventory. These imports are dynamic at execution time, but...
• Bundling: ... the bundling determines and executes all imports before the code is assembled. Then, a bundled, optimized version of the app source code is produced.

Changing how the Controller Works

Before the app is started, we bootstrap the inventory with the following function.

export default function bootstrap (rootDir) {
  const inventory = { pages: [], components: [], actions: [] }
  Object.keys(inventory).forEach(entity => {
    const files = fs.readdirSync(path.join(rootDir, entity), {
      withFileTypes: true
    })
    const fullPath = path.join(path.resolve(rootDir), entity)
    files.forEach(file =>
      inventory[entity].push(path.join(fullPath, file.name))
    )
  })
  return inventory
}

This functions traverses, in the rootDir, the subdirectories /pages, /components and /actions, and collects a list of all contained files. The filenames will be stored with their full path to make bundling easier.

Then, the controller.init() uses this inventory to create the internal map objects.

init() {
  this._initMap(Page, 'pages', /Page.js/)
  this._initMap(Action, 'actions', /Action.js/)
  this._initMap(Component, 'components', /Component.js/)
}

_initMap (parentClass, mapType, pattern) {
  this.inventory[mapType].forEach(async filePath => {
    try {
      if (!filePath.match(pattern)) {
        throw new Error()
      }
      const name = filePath
        .split('/')
        .pop()
        .replace(pattern, '')
      const clazz = (await import(`${filePath}`)).default

      if (clazz.prototype instanceof parentClass) {
        if (parentClass === Page) {
          const route = `/${name.replace(/([a-zA-Z])(?=[A-Z])/g, '$1-').toLowerCase()}`
          this[mapType].set(name, { route, clazz })
        } else {
          this[mapType].set(name, { clazz })
        }
      } else {
        throw new Error()
      }
    } catch (e) {
      console.error(e)
      throw new (class EntityLoadError extends Error {
        message = `Entity ${parentClass.name} from path ${filePath} could not be loaded`
      })()
    }
  })
}

This method traverses each directory, and checks each file. If the file does not have a name that ends with its type, or if the export does not contain a class of the requested, it is not imported, but an error is thrown. If all checks are passed, class object is dynamically loaded and added to the corresponding map.

Building & Bundling Commands

With these changed, SPAC apps can be built. For convenience, the necessary steps are exposed as npm commands.

Building the app consists for two steps. First, the bootstrap command creates the inventory files. This step needs to happen on the machine at which SPAC is installed, because it requires some core NodeJS libraries that cannot be imported or transpiled to the browser. Second, the build command will initiate bundling the complete application code. You need to transfer the bundled file to a web server, or for local development you can use the dev command which starts a snowpack build-in server.

Snowpack Config File

Snowpack is a bundler that comes with a robust default configuration. You can change several aspects with a snowpack.config.js file. In order to control the final layout, and details of the bundling process, I use the following config file.

module.exports = {
  mount: {
    public: '/',
    src: '/src'
  },
  devOptions: {
    bundle: true,
    clean: true
  },
  installOptions: {
    treeshake: true
  },
  buildOptions: {
    out: 'build',
    clean: true,
    metaDir: '/core',
    webModulesUrl: '/lib'
  }
}

The config file is separated into four sections with the following meaning.

• mount: Configure additional folders to be served in your build, where src is the absolute path in your project, and public the folder to which these files will be copied
• devOptions: Control how the dev command works, here I add options to clean the cache and to use the bundled version of the code. This option is important to save you valuable time when your builds are not working - figure out the errors rather earlier.
• installOptions: During the bundling step, I use treeshake to eliminate redundant and dead code in the application and libraries
• buildOptions: The bundled source code is copied to out, but before new files are copied, everything is deleted with the clean option. Then, all additional libraries are installed at the webModulesUrl folder, and the metaDir defines where snowpack modules will be installed.

When using all of the above options, the build directory has the following structure:

build
├── core
│   └── env.js
├── img
│   └── favicon.ico
├── index.html
├── lib
│   ├── import-map.json
│   └── spac.js
├── src
│   ├── actions
│   │   ├── SearchApiAction.js
│   │   ├── ...
│   ├── components
│   │   ├── ApiSearchBarComponent.js
│   │   ├── ...
│   ├── globals
│   │   └── icons.js
│   ├── index.js
│   ├── inventory.json
│   ├── inventory.json.proxy.js
│   └── pages
│       ├── IndexPage.js
│       ├── ...
├── ...
└── style.css

Snowpack takes care to minify the bundled source code, but does not obfuscate the code - for this, you need to configures the @snowpack/webpack-plugin and provide a custom webpack configuration.

The Final Frontier: Caching Dynamic Imports

During the development, I figured out that bundling does not pre-load the dynamic imports. Instead, they are issued at runtime, against the webserver providing your application. This does not impair functionality, but results in several more HTTP request from the users’ browser back to the server. To prevent this, my current idea is to modify the bootstrap command with a preload option that will store the file content of pages, components and actions. Then, at runtime, these strings would be evaluated. How? JavaScript supports running strings as code with eval(), but this has severe security implications as detailed in mdn documentation. There are alternatives to consider. But for the time being, this is an ongoing development.

Conclusion

This article covered an interesting development phase of SPAC: Figuring out how to serve the application from a web server to the browser. I discovered the important syntactical differences of NodejS modules, present on the machine on which you develop your application, and commonJS modules that run in your browser. Also, dynamic imports of files differ: On your machine, the file system is accessed, inside the browser, HTTP requests are made. This difference is also solved through the bundler. I continued to explain convenient commands that will bootstrap, build and deploy the application code, and finally detailed the bundlers configuration options.

• Javascript
• Framework
• Spac_framework