Tom's Blog

emscripten - got gamekit running

Thomas Trocha -

I finally got gamekit to compile and run in the browser. Emscripten magically converted it for me. Thx,a lot :D At least it seems so. Not tested that well.

That state now:

  • I can compile gamekit.
  • There is a cmake-option to create a html-file and set a virtual-folder. This folder will be packaged and set along with the executable and automatically called in the executable.
  • Once I have a compiled js-binary I want to have to possiblity to just package without the need to recompile everytime. This is done using $EMSCRIPTEN/tools/file_packager.js
python $EMSCRIPTEN/tools/file_packager.py [data-file] --js-output=[loader-file] --preload [folder to be packaged]

e.g.:
python $EMSCRIPTEN/tools/file_packager.py project.data --js-output=loader.js --preload .

file_packager.py will always pack relative to current folder. So in the example it packs all files and subdirectories into the project.data-file and creates a loader.js that is actually loading this data-file. This loader.js needs to be called before our engine starts.

Here more details about the file_packager.py-call:

Usage:
  file_packager.py TARGET [--preload A [B..]] [--embed C [D..]] [--exclude E [F..]] [--crunch[=X]] [--js-output=OUTPUT.js] [--no-force] [--use-preload-cache] [--no-heap-copy] [--separate-metadata] [--lz4] [--use-preload-plugins]
  --preload  ,
  --embed    See emcc --help for more details on those options.
  --no-closure In general, the file packager emits closure compiler-compatible code, which requires an eval().
               With this flag passed, we avoid emitting the eval. emcc passes this flag by default whenever
               it knows that closure is not run.
  --crunch=X Will compress dxt files to crn with quality level X. The crunch commandline tool must be present
             and CRUNCH should be defined in ~/.emscripten that points to it. JS crunch decompressing code will
             be added to convert the crn to dds in the browser.
             crunch-worker.js will be generated in the current directory. You should include that file when
             packaging your site.
             DDS files will not be crunched if the .crn is more recent than the .dds. This prevents a lot of
             unneeded computation.
  --js-output=FILE Writes output in FILE, if not specified, standard output is used.
  --no-force Don't create output if no valid input file is specified.
  --use-preload-cache Stores package in IndexedDB so that subsequent loads don't need to do XHR. Checks package version.
  --no-heap-copy If specified, the preloaded filesystem is not copied inside the Emscripten HEAP, but kept in a separate typed array outside it.
                 The default, if this is not specified, is to embed the VFS inside the HEAP, so that mmap()ing files in it is a no-op.
                 Passing this flag optimizes for fread() usage, omitting it optimizes for mmap() usage.
  --separate-metadata Stores package metadata separately. Only applicable when preloading and js-output file is specified.
  --lz4 Uses LZ4. This compresses the data using LZ4 when this utility is run, then the client decompresses chunks on the fly, avoiding storing
        the entire decompressed data in memory at once. See LZ4 in src/settings.js, you must build the main program with that flag.
  --use-preload-plugins Tells the file packager to run preload plugins on the files as they are loaded. This performs tasks like decoding images
                        and audio using the browser's codecs.

How to add the generated code?
I took the generated html as a base. So just search for your created js-executable and just before that include:

var loader = document.createElement('script');
loader.src = "loader.js";
document.body.appendChild(loader);

Afterwards It should look like this:

(function() {
  var memoryInitializer = 'bbkit.js.mem';
...
})();

var loader = document.createElement('script');
loader.src = "loader.js";
document.body.appendChild(loader);

var script = document.createElement('script');
script.src = "yourexe.js";
document.body.appendChild(script);

So you ensure that the package is processed before the engine starts.
It is important to mention that the memoryInitializer(see code above) needs to be set properly. The bbkit.js.mem-file is created on compilation. If you choose to create an html-file the name will be bbkit.html.mem and bbkit.js.mem otherwise. Just for info.

I'm really glad I kept pushing myself to go on with emscripten. I never expected to come to such a state(still not much testing). Actually it is only possible because there is now (experimental) pthread-support. One year ago I had to take out all thread-based logic.

Here are some thoughts that kept in my mind:

# Fetch the latest registry of available tools.
./emsdk update

# Download and install the latest SDK tools.
./emsdk install latest

# Set up the compiler configuration to point to the "latest" SDK.
./emsdk activate latest

# Linux/Mac OS X only: Set the current Emscripten path
source ./emsdk_env.sh
  • In Cmake use the current toolchain-file: $EMSCRIPTEN/emscripten/[brance]/CMake/Packages/Platform
  • In order to work with pthreads you have to set -s USE_PTHREADS=1 to cxx-flags and c-flags
  • when using pthreads you cannot use automatic memory growth, so you need to set a limit for the space that needs to be reserved. To do this use something like this -s TOTAL_MEMORY=1536870912 (in bytes) here 1.5gb again for C- and CXX-flags
  1. To get pthreads work you have to meet some requirements:
    Both Firefox and Chrome ship with prototype implementations of the proposal; these are largely compatible.
  • The feature is enabled by default in Firefox Nightly; starting with Firefox 46, users of Developer Edition, Aurora, Beta, and Release can visit about:config and set the option javascript.options.shared_memory to true.

  • The feature is off by default in Chrome, but can be enabled by passing the command line options --js-flags=--harmony-sharedarraybuffer and --enable-blink-feature=SharedArrayBuffer. (Known to work in Chrome 48.)

  • From time to time I get that error:

ERROR:root:Emscripten, llvm and clang repo versions do not match, this is dangerous (1.36.0, 1.36.14, 1.36.14)
ERROR:root:Make sure to use the same branch in each repo, and to be up-to-date on each. See http://kripken.github.io/emscripten-site/docs/building_from_source/LLVM-Backend.html

Not sure why because I activated emsdk the right way. Just start 'make' again and it works. At least here for me.

Ok, and whoever came till this point, here is a little sample of gamekit running in the browser. (About 30MB! Got it to shrink a little bit ;) )

Refs:
https://kripken.github.io/emscripten-site/docs/getting_started/downloads.html
https://kripken.github.io/emscripten-site/docs/porting/files/packaging_files.html
https://kripken.github.io/emscripten-site/docs/porting/pthreads.html
https://github.com/tc39/ecmascript_sharedmem