<< November 2016 | Home | January 2017 >>


Still working on the ZeroMQ-Front. And I am still convinced, now that I finished most of the guide. Actually I did plan to use JeroMQ but this pure Java ZeroMQ-Port seems to not have implemented security,yet.

Therefore I switched to the jni-Binding of libzmq which have everything implemented but the security (CURVE) that I wanted to use. :D The source-code says:

                if (self.mechanism.equals("PLAIN")) {
                    self.username = request.popString();
                    self.password = request.popString();
                } else if (self.mechanism.equals("CURVE")) {
                    // TODO: Handle CURVE authentication
                } else if (self.mechanism.equals("GSSAPI")) {
                    self.principal = request.popString();

Nothing that shocks me too much. I'm not afraid to make my hands dirty. Therefore my plan was to first try security using C and then look if it is too complicated to get this mapped via jni.

Well, I found a wonderful article that describes everything about CURVE-based security with ZeroMQ,but for API v2 which is of course not compatible to API v3. ;) Took me some time to port those examples to compile with the current version of CZMQ, which is a 'High-level C binding for ZMQ'. Actually I found those files from the article as examples in the CZMQ-Git-repository and made a pullrequest for this. Let's see, what they have to say ;)
EDIT: Funny thing, while writing they accepted the pullrequest. Soooo proud ;)

Now I have two options:

  1. Write the proper mapping so I can get it work with JZMQ (the java-jni-binding)
  2. Write a function that creates everything I want and make it accessable via JNI.

But this have to wait till tomorrow. Now a little bit gamekit-fun :D

EDIT2: Ok, found out that JZMQ actually do have encryption built in and with this ZAP-Request you only have the final say if you are willing to allow or disallow a connection. Atm, I'm implementing CertStores for JZMQ, so only known clients are accepted as connecting clients. But this will be another post.

Tags : , , ,

ZeroMQ here we go

At the moment I'm working on a more distributed project. One central thing is that I want a local-server to be control from the main-server that is located somewhere else. (e.g. the internet) I'm using thrift-services and want on the one side to be able that the local-server can call the service-layer of the main-server and on the other side I want the main-server to be able to call the service-layer on the local-server.
'Real' clients connect via a separate port to the server's Client-Service-Layer.

I actually have a solution. I share the socket that is created by the local server when it connects to the main-server.

Meaning for the main-server:

  • incoming-data goes to the main-server-service-implementation
  • outgoing-data is mapped to a client of the local-server-thrift-client

The problem with this approach, you are forced to use oneway-calls in your thrift-service, which is a bit hacky if you want to give back information or want to throw exceptions. Otherwise you would have to label the traffic so everything is sent to the right requesting calls. (EDIT: Another approach could have been to open a second connection from client to server and to use on for the incoming and the other for the outgoing traffic )


Well, the oneway-call-approach works, nonetheless I have had the feeling that it is a bit too hacky. And since ZeroMQ is on my radar for some time already I throw the thrift-transport to the trash and create one based on ZeroMQ. Good chance to use it in a real project instead of just doing some samples or tutorials.

Btw, the ZeroMQ-guide is very good to read and quite good to follow and even funny ;) Here some quotes:

  • ...TCP has a long timeout (30 minutes or so), that means that it can be impossible to know whether a peer has died, been disconnected, or gone on a weekend to Prague with a case of vodka, a redhead, and a large expense account....
  • ...Let's see an example that pushes out weather updates consisting of a zip code, temperature, and relative humidity. We'll generate random values, just like the real weather stations do....
  • ...What it means is that it's safe to repeat an operation. Checking the clock is idempotent. Lending ones credit card to ones children is not...
    Some read about thrift bidirectional-connection

PS: Btw, this is the first blog-post I write with my markdown-plugin for pebble. (This should be a post of its own)

Tags : , , ,

Build-Tool 'FUN'

Everytime I start with a new project I want to make it right from the start. And with this comes to write a proper build-file. No matter if this is cmake,ant,maven,gradle,premake... I end up thinking "Why the 'funk' is that so complicated and why is this not working what should obviously work...." I guess every new build-system started to make things easier but in the end it will end up in being an unintuitive mess.

Ok, I have to be honest. I'm not writing those build-files too often and have the "just do it"-approach, but hey. That have to be much simpler. Since I'm rotating between cmake, maven and ant I just forget from one project to another how things work... I really have to think about this and make things easier :)

Gamekit - Tutorials

Just started to create tutorials for getting started with "gamekit playground" which is actually the name for my current fragile buggy version of gamekit in which I included lots(?) of new features including an easy to use nodetree-mapping from blender to gamekit.

For this I need to change some templates for the pebble-blog which turns out to be more flexible than thought. No problem to set custom templates for static pages.

Have a look at the Gamekit-Playground-Pages

bash - scripting

Now that all platforms (beside the apple ones) can be built on linux, I want a fabulous build-all script, that automagically creates a folder with all runtimes. Time to refresh my bash-basics.

And to say it from the start: Spaces do matter!!

  • How to set a variable

      # GOOD:
      # WRONG:
      varname = "fortuna"
  • Show argument count

    echo number of arguments: $#
    echo script itself: $0
    echo arg1:$1 arg2:$2 arg3:$3  ...
  • basic conditionals for number comparison with if-elif-else-construct using some checks: -eq, -ne, -lt, -le, -gt, -ge

    if [[ $# -eq 0 ]]; then
      echo "NO ARGS"
    elif [[ $# -eq 1 ]]; then
      echo "ONE ARGUMENT"
      echo "$# Arguments"
    # not equals
    if [[ $# -ne 1 ]]; then
      echo "arg-count != 1"
      echo "arg-count == 1"
    # check if a directory does NOT exist
    if [[ ! -d "linux" ]]; then
      echo "linux-directory does not exist"

    For string-comparison use: =,!=

  • Some useful directory-magic:

    # get the directory from which the script is called
    # get the directory where the script is located (relative)
    REL_BASEDIR=$(dirname $0)
    # get the directory where the script is located (absolute)
    ABS_BASEDIR=$( cd $(dirname $0) ; pwd )
    # get the directory where the script is located (absolute with resolved symlinks)
    ABS_BASEDIR_RESOLVED=$( cd $(dirname $0) ; pwd -P )
  • Check if something went wrong (e.g. with cmake).

    cmake -DCMAKE_BUILD_TYPE=Release .. || {
      echo "Something went wrong with CMAKE! Exiting..."
      exit 1;
    echo "Everything went fine!"
  • exit the script:

    # just exit with code 0
    # give it a special exit-code you can use on the shell
    exit 95
    There are some special exit-codes
  • creating functions:

    function myEcho {
        echo $1
    myEcho Hello
  • Prevent a command to output errors

    # if a.txt is not present do not output any error
    rm a.txt 2>/dev/null

* Bash Conditionals http://www.gnu.org/software/bash/manual/bash.html#Bash-Conditional-Expressions
* Testing exit values https://sanctum.geek.nz/arabesque/testing-exit-values-bash/
* Functions http://tldp.org/HOWTO/Bash-Prog-Intro-HOWTO-8.html

emscripten - got gamekit running

I finally got gamekit to compile and run in the browser. Emscripten magically converted it for me. Thx,a lot :D At least it seems so. Not tested that well.

That state now:

  • I can compile gamekit.
  • There is a cmake-option to create a html-file and set a virtual-folder. This folder will be packaged and set along with the executable and automatically called in the executable.
  • Once I have a compiled js-binary I want to have to possiblity to just package without the need to recompile everytime. This is done using $EMSCRIPTEN/tools/file_packager.js
python $EMSCRIPTEN/tools/file_packager.py [data-file] --js-output=[loader-file] --preload [folder to be packaged]

python $EMSCRIPTEN/tools/file_packager.py project.data --js-output=loader.js --preload .

file_packager.py will always pack relative to current folder. So in the example it packs all files and subdirectories into the project.data-file and creates a loader.js that is actually loading this data-file. This loader.js needs to be called before our engine starts.

Here more details about the file_packager.py-call:

  file_packager.py TARGET [--preload A [B..]] [--embed C [D..]] [--exclude E [F..]] [--crunch[=X]] [--js-output=OUTPUT.js] [--no-force] [--use-preload-cache] [--no-heap-copy] [--separate-metadata] [--lz4] [--use-preload-plugins]
  --preload  ,
  --embed    See emcc --help for more details on those options.
  --no-closure In general, the file packager emits closure compiler-compatible code, which requires an eval().
               With this flag passed, we avoid emitting the eval. emcc passes this flag by default whenever
               it knows that closure is not run.
  --crunch=X Will compress dxt files to crn with quality level X. The crunch commandline tool must be present
             and CRUNCH should be defined in ~/.emscripten that points to it. JS crunch decompressing code will
             be added to convert the crn to dds in the browser.
             crunch-worker.js will be generated in the current directory. You should include that file when
             packaging your site.
             DDS files will not be crunched if the .crn is more recent than the .dds. This prevents a lot of
             unneeded computation.
  --js-output=FILE Writes output in FILE, if not specified, standard output is used.
  --no-force Don't create output if no valid input file is specified.
  --use-preload-cache Stores package in IndexedDB so that subsequent loads don't need to do XHR. Checks package version.
  --no-heap-copy If specified, the preloaded filesystem is not copied inside the Emscripten HEAP, but kept in a separate typed array outside it.
                 The default, if this is not specified, is to embed the VFS inside the HEAP, so that mmap()ing files in it is a no-op.
                 Passing this flag optimizes for fread() usage, omitting it optimizes for mmap() usage.
  --separate-metadata Stores package metadata separately. Only applicable when preloading and js-output file is specified.
  --lz4 Uses LZ4. This compresses the data using LZ4 when this utility is run, then the client decompresses chunks on the fly, avoiding storing
        the entire decompressed data in memory at once. See LZ4 in src/settings.js, you must build the main program with that flag.
  --use-preload-plugins Tells the file packager to run preload plugins on the files as they are loaded. This performs tasks like decoding images
                        and audio using the browser's codecs.

How to add the generated code?
I took the generated html as a base. So just search for your created js-executable and just before that include:

var loader = document.createElement('script');
loader.src = "loader.js";

Afterwards It should look like this:

(function() {
  var memoryInitializer = 'bbkit.js.mem';

var loader = document.createElement('script');
loader.src = "loader.js";

var script = document.createElement('script');
script.src = "yourexe.js";

So you ensure that the package is processed before the engine starts.
It is important to mention that the memoryInitializer(see code above) needs to be set properly. The bbkit.js.mem-file is created on compilation. If you choose to create an html-file the name will be bbkit.html.mem and bbkit.js.mem otherwise. Just for info.

I'm really glad I kept pushing myself to go on with emscripten. I never expected to come to such a state(still not much testing). Actually it is only possible because there is now (experimental) pthread-support. One year ago I had to take out all thread-based logic.

Here are some thoughts that kept in my mind:

# Fetch the latest registry of available tools.
./emsdk update

# Download and install the latest SDK tools.
./emsdk install latest

# Set up the compiler configuration to point to the "latest" SDK.
./emsdk activate latest

# Linux/Mac OS X only: Set the current Emscripten path
source ./emsdk_env.sh
  • In Cmake use the current toolchain-file: $EMSCRIPTEN/emscripten/[brance]/CMake/Packages/Platform
  • In order to work with pthreads you have to set -s USE_PTHREADS=1 to cxx-flags and c-flags
  • when using pthreads you cannot use automatic memory growth, so you need to set a limit for the space that needs to be reserved. To do this use something like this -s TOTAL_MEMORY=1536870912 (in bytes) here 1.5gb again for C- and CXX-flags
  1. To get pthreads work you have to meet some requirements:
    Both Firefox and Chrome ship with prototype implementations of the proposal; these are largely compatible.
  • The feature is enabled by default in Firefox Nightly; starting with Firefox 46, users of Developer Edition, Aurora, Beta, and Release can visit about:config and set the option javascript.options.shared_memory to true.
  • The feature is off by default in Chrome, but can be enabled by passing the command line options --js-flags=--harmony-sharedarraybuffer and --enable-blink-feature=SharedArrayBuffer. (Known to work in Chrome 48.)

  • From time to time I get that error:

ERROR:root:Emscripten, llvm and clang repo versions do not match, this is dangerous (1.36.0, 1.36.14, 1.36.14)
ERROR:root:Make sure to use the same branch in each repo, and to be up-to-date on each. See http://kripken.github.io/emscripten-site/docs/building_from_source/LLVM-Backend.html

Not sure why because I activated emsdk the right way. Just start 'make' again and it works. At least here for me.

Ok, and whoever came till this point, here is a little sample of gamekit running in the browser. (About 30MB! Got it to shrink a little bit ;) )


emscripten reloaded

At the moment I'm in the process to get gamekit being compilable on the main platforms. So for now I got linux,win32(cross compiled with mxe(mingw) on linux),android,blackberry playbook(my precious thing),blackberry bb10 (can create a package but crashes in the emulator when ogre tries to get the opengl-extensions. I decided to skip for now, as this platform is already dead. Sad but true).

So now emscripten to have gamekit running in the browser (again!?). I was already working on this about one year ago and I kind of got it to work. I still know that I spent plenty of time getting the reddwarf-server accept emscriptens socket=>websocket=>server-Mapping running.

Now I actually have to start from zero. I could have sworn I wrote a howto, but damned me it seems it did not.

To make a long story short, I finally came to a point where I could compile and link the whole gamekit-project but when it came to create a website it crashed. There is a hint that I got a mismatch in my emscripten<=>llvm<=>clang versions.

Now I'm updating the emscripten-library the right way

hooary Once I took the CMake-Toolchain file that is packed with the library and not my outdated one I got it to compile. It also generates a html-file that I just need to start. Gamekit is starting but then hangs somewhere.

I am using the new pthreads-feature and I guess there might be a problem. For now I'm glad I came to this point!

At least the Input is working, so that is actually not bad. The js-file is about 300M ;) but I have turned off almost any optimization. There is still some room for improvement!

Now I'm trying to work on the top notch emscripten version.... Don't need a heater atm. Compling the thrid emscripten-sdk from scratch now ;)

xtext - create update site with your plugin

If developing with xtext you usually test your DSL in an runtimeEclipse-Version that you start from within eclipse. Once you have a stable state of your plugin, you will want to make it available to the public.

Do this:

  • In Eclipse: XText generates 3+ Folders for you at the beginning. Locate the one with the .sdk-postfix. "e.g. org.tt.mysdl.sdk"

  • Right-Click on Project => NEW => Plug-in Development => Catergory Definition

  • In the category.xml-Editor => New Category (give it a random name. e.g. MyPlugins )

  • Select the category.

  • Add Feature => Locate you .sdk and add it

  • Right-Click on the feature.xml-file => Export => Plug-in Development=>Deployable features=>NEXT

  • Choose a destination-folder, in which the eclipse update-site will be created

  • Options => Check Categorize Repository and browse to the category.xml we just created.

  • Hit FINISH

Now we successfully create a local eclipse update-site.

If you now want to install your plugin in eclipse:

  • HELP => Install New Software => ADD => Give it a name (e.g. MyPlugins-LOCAL) and press "Local..."
  • Select your Folder
  • You should see your category which has one child - your plugin
  • Select your Plugin => Next => Finish


  1. It seems that those plugin-sites are cached heavily. So if some changes are not getting visible, restart eclipse.
  2. If you already installed one plugin somewhere, and change something, do not forget to change the version-number
Tags : , ,

xtext - export dsl as runnable jar

If you write an xtext-dsl you get a great editor and generator inside of eclipse. But there are times you want the generator to generate the code outside of eclipse. e.g. for using in a automated workflow.

I'm not sure what new options were introduced since XText 2.8 but I guess it is worth mentioning nonetheless.

Here are the steps to take:

  1. Make sure to add generateJavaMain = true to your DSL's mwe2-file:

    fragment = generator.GeneratorFragment auto-inject {
      generateJavaMain = true

    Next time you rerun mwe2 (right-click on your mwe2-file => run-as mwe2) a file called Main.java will be generated in the Generator-Folder

  2. Create a Run-Configuration to start the generated Main.java

  3. Export your dsl as runable-jar:
  • right-click on your dsl-project => export=>Java=>Runnable Jar File
  • Select the Launch-Configuration that is starting the Main.java
  • Select the output-Folder
  • Using the Save as Ant-Script is optional but becomes handy if you are testing a lot.
    Once you have create this you can call it to pack the jar again. (It will not build, this is usually done automatically by eclipse) e.g. like this

      ant -f packjar.xml
  1. Your basically done. You can now create your generated output by calling:

      java --jar mydsl-runnable.jar input.mydsl

This will usually create the output in a src-gen-folder. If you want to override this behaviour you can use OutputConfigurations.


  • To change the outputFolder you can change the outputPath in the Main.java:

      // configure and start the generator
  • More advanced are output-configurations. In this you can e.g. tell the dsl's filesystem how to behave if a file is already generated. In one project I wanted some files to overwrite old versions and some files to be created once and never overwrite those (e.g. because I want the user to add their logic in there). Here is an example. You have to replace the setOutputPath-Part with those lines in Main.java:

  String outputFolder = "./out";

  OutputConfiguration defaultOutput = new OutputConfiguration(IFileSystemAccess.DEFAULT_OUTPUT);
  defaultOutput.setDescription("Output Folder");
  // tell the fsa to override(!) existing resources!!

  OutputConfiguration genonceOutput = new OutputConfiguration("gen-once");
  genonceOutput.setDescription("Generate Once Output Folder");
  // tell the fsa to NOT override existing resources!!

  Map<String,OutputConfiguration> fsaConfs = new HashMap<>();
  fsaConfs.put(IFileSystemAccess.DEFAULT_OUTPUT, defaultOutput);


You can access the different Configurations like this:

  fsa.generateFile(String fileName,String outputConfigurationName,CharSequence contents);

Obviously this all makes only sense if you also created something like this in your eclipse-runtime.

Sry, for the redundant code here. You actually should have one solution that get's called at an appropriate time. Ok, here we go:

  • Create an xtend-class 'CustomOutputConfigurationProvider.xtend' in your dsl's main-folder with this content (make sure to adapt package and other things that might differ):
  package org.tt.gen

  import java.util.Set
  import org.eclipse.xtext.generator.OutputConfiguration
  import org.eclipse.xtext.generator.IOutputConfigurationProvider
  import org.eclipse.xtext.generator.IFileSystemAccess

  public class CustomOutputConfigurationProvider implements IOutputConfigurationProvider {

    public static final String GEN_ONCE_OUTPUT = "gen-once";

     * @return a set of {@link OutputConfiguration} available for the generator
     override def Set<OutputConfiguration> getOutputConfigurations() {
      val OutputConfiguration defaultOutput = new OutputConfiguration(IFileSystemAccess.DEFAULT_OUTPUT);
      defaultOutput.setDescription("Output Folder");

      val OutputConfiguration readonlyOutput = new OutputConfiguration(GEN_ONCE_OUTPUT);
      readonlyOutput.setDescription("Read-only Output Folder");
      return newHashSet(defaultOutput, readonlyOutput);
  • Now we need to tell the runtime-version to use our new custom outputconfiguration-provider. (Viva dependency injection!!).
    Add following lines in your DSL's runtime-module-class' configure-method that is located in your DSL's main-folder:


That should be it.

Tags : ,
<< November 2016 | Home | January 2017 >>