<< Previous | Home | Next >>

Build-Tool 'FUN'

Everytime I start with a new project I want to make it right from the start. And with this comes to write a proper build-file. No matter if this is cmake,ant,maven,gradle,premake... I end up thinking "Why the 'funk' is that so complicated and why is this not working what should obviously work...." I guess every new build-system started to make things easier but in the end it will end up in being an unintuitive mess.

Ok, I have to be honest. I'm not writing those build-files too often and have the "just do it"-approach, but hey. That have to be much simpler. Since I'm rotating between cmake, maven and ant I just forget from one project to another how things work... I really have to think about this and make things easier :)

Export this post as PDF document  Export this post to PDF document

Social Bookmarks :  Add this post to Slashdot    Add this post to Digg    Add this post to Reddit    Add this post to Delicious    Add this post to Stumble it    Add this post to Google    Add this post to Technorati    Add this post to Bloglines    Add this post to Facebook    Add this post to Furl    Add this post to Windows Live    Add this post to Yahoo!

Gamekit - Tutorials

Just started to create tutorials for getting started with "gamekit playground" which is actually the name for my current fragile buggy version of gamekit in which I included lots(?) of new features including an easy to use nodetree-mapping from blender to gamekit.

For this I need to change some templates for the pebble-blog which turns out to be more flexible than thought. No problem to set custom templates for static pages.

Have a look at the Gamekit-Playground-Pages

Export this post as PDF document  Export this post to PDF document

Social Bookmarks :  Add this post to Slashdot    Add this post to Digg    Add this post to Reddit    Add this post to Delicious    Add this post to Stumble it    Add this post to Google    Add this post to Technorati    Add this post to Bloglines    Add this post to Facebook    Add this post to Furl    Add this post to Windows Live    Add this post to Yahoo!

bash - scripting

Now that all platforms (beside the apple ones) can be built on linux, I want a fabulous build-all script, that automagically creates a folder with all runtimes. Time to refresh my bash-basics.

And to say it from the start: Spaces do matter!!

  • How to set a variable

      # GOOD:
      varname="fortuna"
      # WRONG:
      varname = "fortuna"
  • Show argument count

    #!/bin/bash
    echo number of arguments: $#
    echo script itself: $0
    echo arg1:$1 arg2:$2 arg3:$3  ...
  • basic conditionals for number comparison with if-elif-else-construct using some checks: -eq, -ne, -lt, -le, -gt, -ge

    #!/bin/bash
    
    #equals
    if [[ $# -eq 0 ]]; then
      echo "NO ARGS"
    elif [[ $# -eq 1 ]]; then
      echo "ONE ARGUMENT"
    else
      echo "$# Arguments"
    fi
    
    # not equals
    if [[ $# -ne 1 ]]; then
      echo "arg-count != 1"
    else
      echo "arg-count == 1"
    fi
    
    # check if a directory does NOT exist
    if [[ ! -d "linux" ]]; then
      echo "linux-directory does not exist"
    fi

    For string-comparison use: =,!=

  • Some useful directory-magic:

    # get the directory from which the script is called
    CURRENT_DIR=$PWD
    
    # get the directory where the script is located (relative)
    REL_BASEDIR=$(dirname $0)
    
    # get the directory where the script is located (absolute)
    ABS_BASEDIR=$( cd $(dirname $0) ; pwd )
    
    # get the directory where the script is located (absolute with resolved symlinks)
    ABS_BASEDIR_RESOLVED=$( cd $(dirname $0) ; pwd -P )
  • Check if something went wrong (e.g. with cmake).

    cmake -DCMAKE_BUILD_TYPE=Release .. || {
      echo "Something went wrong with CMAKE! Exiting..."
      exit 1;
    }
    
    echo "Everything went fine!"
  • exit the script:

    # just exit with code 0
    exit
    
    # give it a special exit-code you can use on the shell
    exit 95
    There are some special exit-codes
  • creating functions:

    #!/bin/bash
    function myEcho {
        echo $1
    }  
    myEcho Hello
  • Prevent a command to output errors

    # if a.txt is not present do not output any error
    rm a.txt 2>/dev/null

Links:
* Bash Conditionals http://www.gnu.org/software/bash/manual/bash.html#Bash-Conditional-Expressions
* Testing exit values https://sanctum.geek.nz/arabesque/testing-exit-values-bash/
* Functions http://tldp.org/HOWTO/Bash-Prog-Intro-HOWTO-8.html

Export this post as PDF document  Export this post to PDF document

Social Bookmarks :  Add this post to Slashdot    Add this post to Digg    Add this post to Reddit    Add this post to Delicious    Add this post to Stumble it    Add this post to Google    Add this post to Technorati    Add this post to Bloglines    Add this post to Facebook    Add this post to Furl    Add this post to Windows Live    Add this post to Yahoo!

emscripten - got gamekit running

I finally got gamekit to compile and run in the browser. Emscripten magically converted it for me. Thx,a lot :D At least it seems so. Not tested that well.

That state now:

  • I can compile gamekit.
  • There is a cmake-option to create a html-file and set a virtual-folder. This folder will be packaged and set along with the executable and automatically called in the executable.
  • Once I have a compiled js-binary I want to have to possiblity to just package without the need to recompile everytime. This is done using $EMSCRIPTEN/tools/file_packager.js
python $EMSCRIPTEN/tools/file_packager.py [data-file] --js-output=[loader-file] --preload [folder to be packaged]

e.g.:
python $EMSCRIPTEN/tools/file_packager.py project.data --js-output=loader.js --preload .

file_packager.py will always pack relative to current folder. So in the example it packs all files and subdirectories into the project.data-file and creates a loader.js that is actually loading this data-file. This loader.js needs to be called before our engine starts.

Here more details about the file_packager.py-call:

Usage:
  file_packager.py TARGET [--preload A [B..]] [--embed C [D..]] [--exclude E [F..]] [--crunch[=X]] [--js-output=OUTPUT.js] [--no-force] [--use-preload-cache] [--no-heap-copy] [--separate-metadata] [--lz4] [--use-preload-plugins]
  --preload  ,
  --embed    See emcc --help for more details on those options.
  --no-closure In general, the file packager emits closure compiler-compatible code, which requires an eval().
               With this flag passed, we avoid emitting the eval. emcc passes this flag by default whenever
               it knows that closure is not run.
  --crunch=X Will compress dxt files to crn with quality level X. The crunch commandline tool must be present
             and CRUNCH should be defined in ~/.emscripten that points to it. JS crunch decompressing code will
             be added to convert the crn to dds in the browser.
             crunch-worker.js will be generated in the current directory. You should include that file when
             packaging your site.
             DDS files will not be crunched if the .crn is more recent than the .dds. This prevents a lot of
             unneeded computation.
  --js-output=FILE Writes output in FILE, if not specified, standard output is used.
  --no-force Don't create output if no valid input file is specified.
  --use-preload-cache Stores package in IndexedDB so that subsequent loads don't need to do XHR. Checks package version.
  --no-heap-copy If specified, the preloaded filesystem is not copied inside the Emscripten HEAP, but kept in a separate typed array outside it.
                 The default, if this is not specified, is to embed the VFS inside the HEAP, so that mmap()ing files in it is a no-op.
                 Passing this flag optimizes for fread() usage, omitting it optimizes for mmap() usage.
  --separate-metadata Stores package metadata separately. Only applicable when preloading and js-output file is specified.
  --lz4 Uses LZ4. This compresses the data using LZ4 when this utility is run, then the client decompresses chunks on the fly, avoiding storing
        the entire decompressed data in memory at once. See LZ4 in src/settings.js, you must build the main program with that flag.
  --use-preload-plugins Tells the file packager to run preload plugins on the files as they are loaded. This performs tasks like decoding images
                        and audio using the browser's codecs.

How to add the generated code?
I took the generated html as a base. So just search for your created js-executable and just before that include:

var loader = document.createElement('script');
loader.src = "loader.js";
document.body.appendChild(loader);

Afterwards It should look like this:

(function() {
  var memoryInitializer = 'bbkit.js.mem';
...
})();

var loader = document.createElement('script');
loader.src = "loader.js";
document.body.appendChild(loader);

var script = document.createElement('script');
script.src = "yourexe.js";
document.body.appendChild(script);

So you ensure that the package is processed before the engine starts.
It is important to mention that the memoryInitializer(see code above) needs to be set properly. The bbkit.js.mem-file is created on compilation. If you choose to create an html-file the name will be bbkit.html.mem and bbkit.js.mem otherwise. Just for info.

I'm really glad I kept pushing myself to go on with emscripten. I never expected to come to such a state(still not much testing). Actually it is only possible because there is now (experimental) pthread-support. One year ago I had to take out all thread-based logic.

Here are some thoughts that kept in my mind:

# Fetch the latest registry of available tools.
./emsdk update

# Download and install the latest SDK tools.
./emsdk install latest

# Set up the compiler configuration to point to the "latest" SDK.
./emsdk activate latest

# Linux/Mac OS X only: Set the current Emscripten path
source ./emsdk_env.sh
  • In Cmake use the current toolchain-file: $EMSCRIPTEN/emscripten/[brance]/CMake/Packages/Platform
  • In order to work with pthreads you have to set -s USE_PTHREADS=1 to cxx-flags and c-flags
  • when using pthreads you cannot use automatic memory growth, so you need to set a limit for the space that needs to be reserved. To do this use something like this -s TOTAL_MEMORY=1536870912 (in bytes) here 1.5gb again for C- and CXX-flags
  1. To get pthreads work you have to meet some requirements:
    Both Firefox and Chrome ship with prototype implementations of the proposal; these are largely compatible.
  • The feature is enabled by default in Firefox Nightly; starting with Firefox 46, users of Developer Edition, Aurora, Beta, and Release can visit about:config and set the option javascript.options.shared_memory to true.
  • The feature is off by default in Chrome, but can be enabled by passing the command line options --js-flags=--harmony-sharedarraybuffer and --enable-blink-feature=SharedArrayBuffer. (Known to work in Chrome 48.)

  • From time to time I get that error:

ERROR:root:Emscripten, llvm and clang repo versions do not match, this is dangerous (1.36.0, 1.36.14, 1.36.14)
ERROR:root:Make sure to use the same branch in each repo, and to be up-to-date on each. See http://kripken.github.io/emscripten-site/docs/building_from_source/LLVM-Backend.html

Not sure why because I activated emsdk the right way. Just start 'make' again and it works. At least here for me.

Ok, and whoever came till this point, here is a little sample of gamekit running in the browser. (About 30MB! Got it to shrink a little bit ;) )

Refs:
https://kripken.github.io/emscripten-site/docs/getting_started/downloads.html
https://kripken.github.io/emscripten-site/docs/porting/files/packaging_files.html
https://kripken.github.io/emscripten-site/docs/porting/pthreads.html
https://github.com/tc39/ecmascript_sharedmem

Export this post as PDF document  Export this post to PDF document

Social Bookmarks :  Add this post to Slashdot    Add this post to Digg    Add this post to Reddit    Add this post to Delicious    Add this post to Stumble it    Add this post to Google    Add this post to Technorati    Add this post to Bloglines    Add this post to Facebook    Add this post to Furl    Add this post to Windows Live    Add this post to Yahoo!

emscripten reloaded

At the moment I'm in the process to get gamekit being compilable on the main platforms. So for now I got linux,win32(cross compiled with mxe(mingw) on linux),android,blackberry playbook(my precious thing),blackberry bb10 (can create a package but crashes in the emulator when ogre tries to get the opengl-extensions. I decided to skip for now, as this platform is already dead. Sad but true).

So now emscripten to have gamekit running in the browser (again!?). I was already working on this about one year ago and I kind of got it to work. I still know that I spent plenty of time getting the reddwarf-server accept emscriptens socket=>websocket=>server-Mapping running.

Now I actually have to start from zero. I could have sworn I wrote a howto, but damned me it seems it did not.

To make a long story short, I finally came to a point where I could compile and link the whole gamekit-project but when it came to create a website it crashed. There is a hint that I got a mismatch in my emscripten<=>llvm<=>clang versions.

Now I'm updating the emscripten-library the right way

hooary Once I took the CMake-Toolchain file that is packed with the library and not my outdated one I got it to compile. It also generates a html-file that I just need to start. Gamekit is starting but then hangs somewhere.

I am using the new pthreads-feature and I guess there might be a problem. For now I'm glad I came to this point!

At least the Input is working, so that is actually not bad. The js-file is about 300M ;) but I have turned off almost any optimization. There is still some room for improvement!

Now I'm trying to work on the top notch emscripten version.... Don't need a heater atm. Compling the thrid emscripten-sdk from scratch now ;)

Export this post as PDF document  Export this post to PDF document

Social Bookmarks :  Add this post to Slashdot    Add this post to Digg    Add this post to Reddit    Add this post to Delicious    Add this post to Stumble it    Add this post to Google    Add this post to Technorati    Add this post to Bloglines    Add this post to Facebook    Add this post to Furl    Add this post to Windows Live    Add this post to Yahoo!

xtext - create update site with your plugin

If developing with xtext you usually test your DSL in an runtimeEclipse-Version that you start from within eclipse. Once you have a stable state of your plugin, you will want to make it available to the public.

Do this:

  • In Eclipse: XText generates 3+ Folders for you at the beginning. Locate the one with the .sdk-postfix. "e.g. org.tt.mysdl.sdk"

  • Right-Click on Project => NEW => Plug-in Development => Catergory Definition

  • In the category.xml-Editor => New Category (give it a random name. e.g. MyPlugins )

  • Select the category.

  • Add Feature => Locate you .sdk and add it

  • Right-Click on the feature.xml-file => Export => Plug-in Development=>Deployable features=>NEXT

  • Choose a destination-folder, in which the eclipse update-site will be created

  • Options => Check Categorize Repository and browse to the category.xml we just created.

  • Hit FINISH

Now we successfully create a local eclipse update-site.

If you now want to install your plugin in eclipse:

  • HELP => Install New Software => ADD => Give it a name (e.g. MyPlugins-LOCAL) and press "Local..."
  • Select your Folder
  • You should see your category which has one child - your plugin
  • Select your Plugin => Next => Finish

Hints:

  1. It seems that those plugin-sites are cached heavily. So if some changes are not getting visible, restart eclipse.
  2. If you already installed one plugin somewhere, and change something, do not forget to change the version-number
Tags : , ,

Export this post as PDF document  Export this post to PDF document

Social Bookmarks :  Add this post to Slashdot    Add this post to Digg    Add this post to Reddit    Add this post to Delicious    Add this post to Stumble it    Add this post to Google    Add this post to Technorati    Add this post to Bloglines    Add this post to Facebook    Add this post to Furl    Add this post to Windows Live    Add this post to Yahoo!

xtext - export dsl as runnable jar

If you write an xtext-dsl you get a great editor and generator inside of eclipse. But there are times you want the generator to generate the code outside of eclipse. e.g. for using in a automated workflow.

I'm not sure what new options were introduced since XText 2.8 but I guess it is worth mentioning nonetheless.

Here are the steps to take:

  1. Make sure to add generateJavaMain = true to your DSL's mwe2-file:

    fragment = generator.GeneratorFragment auto-inject {
      generateJavaMain = true
    }

    Next time you rerun mwe2 (right-click on your mwe2-file => run-as mwe2) a file called Main.java will be generated in the Generator-Folder

  2. Create a Run-Configuration to start the generated Main.java

  3. Export your dsl as runable-jar:
  • right-click on your dsl-project => export=>Java=>Runnable Jar File
  • Select the Launch-Configuration that is starting the Main.java
  • Select the output-Folder
  • Using the Save as Ant-Script is optional but becomes handy if you are testing a lot.
    Once you have create this you can call it to pack the jar again. (It will not build, this is usually done automatically by eclipse) e.g. like this

      ant -f packjar.xml
  1. Your basically done. You can now create your generated output by calling:

      java --jar mydsl-runnable.jar input.mydsl

This will usually create the output in a src-gen-folder. If you want to override this behaviour you can use OutputConfigurations.

Optional:

  • To change the outputFolder you can change the outputPath in the Main.java:

      // configure and start the generator
      fileAccess.setOutputPath("src-gen/");
  • More advanced are output-configurations. In this you can e.g. tell the dsl's filesystem how to behave if a file is already generated. In one project I wanted some files to overwrite old versions and some files to be created once and never overwrite those (e.g. because I want the user to add their logic in there). Here is an example. You have to replace the setOutputPath-Part with those lines in Main.java:

  String outputFolder = "./out";

  OutputConfiguration defaultOutput = new OutputConfiguration(IFileSystemAccess.DEFAULT_OUTPUT);
  defaultOutput.setDescription("Output Folder");
  defaultOutput.setOutputDirectory(outputFolder+"/src-gen");
  // tell the fsa to override(!) existing resources!!
  defaultOutput.setOverrideExistingResources(true);
  defaultOutput.setCreateOutputDirectory(true);
  defaultOutput.setCleanUpDerivedResources(true);
  defaultOutput.setSetDerivedProperty(true);

  OutputConfiguration genonceOutput = new OutputConfiguration("gen-once");
  genonceOutput.setDescription("Generate Once Output Folder");
  genonceOutput.setOutputDirectory(outputFolder+"/src-nodes");
  // tell the fsa to NOT override existing resources!!
  genonceOutput.setOverrideExistingResources(false);
  genonceOutput.setCreateOutputDirectory(true);
  genonceOutput.setCleanUpDerivedResources(false);
  genonceOutput.setSetDerivedProperty(false);

  Map<String,OutputConfiguration> fsaConfs = new HashMap<>();
  fsaConfs.put(IFileSystemAccess.DEFAULT_OUTPUT, defaultOutput);
  fsaConfs.put("gen-once",genonceOutput);

  fileAccess.setOutputConfigurations(fsaConfs);

You can access the different Configurations like this:

  fsa.generateFile(String fileName,String outputConfigurationName,CharSequence contents);

Obviously this all makes only sense if you also created something like this in your eclipse-runtime.

Sry, for the redundant code here. You actually should have one solution that get's called at an appropriate time. Ok, here we go:

  • Create an xtend-class 'CustomOutputConfigurationProvider.xtend' in your dsl's main-folder with this content (make sure to adapt package and other things that might differ):
  package org.tt.gen

  import java.util.Set
  import org.eclipse.xtext.generator.OutputConfiguration
  import org.eclipse.xtext.generator.IOutputConfigurationProvider
  import org.eclipse.xtext.generator.IFileSystemAccess

  public class CustomOutputConfigurationProvider implements IOutputConfigurationProvider {

    public static final String GEN_ONCE_OUTPUT = "gen-once";

    /**
     * @return a set of {@link OutputConfiguration} available for the generator
     */
     override def Set<OutputConfiguration> getOutputConfigurations() {
      val OutputConfiguration defaultOutput = new OutputConfiguration(IFileSystemAccess.DEFAULT_OUTPUT);
      defaultOutput.setDescription("Output Folder");
      defaultOutput.setOutputDirectory("./src-gen");
      defaultOutput.setOverrideExistingResources(true);
      defaultOutput.setCreateOutputDirectory(true);
      defaultOutput.setCleanUpDerivedResources(true);
      defaultOutput.setSetDerivedProperty(true);

      val OutputConfiguration readonlyOutput = new OutputConfiguration(GEN_ONCE_OUTPUT);
      readonlyOutput.setDescription("Read-only Output Folder");
      readonlyOutput.setOutputDirectory("./src-nodes");
      readonlyOutput.setOverrideExistingResources(false);
      readonlyOutput.setCreateOutputDirectory(true);
      readonlyOutput.setCleanUpDerivedResources(false);
      readonlyOutput.setSetDerivedProperty(false);
      return newHashSet(defaultOutput, readonlyOutput);
    }
  }
  • Now we need to tell the runtime-version to use our new custom outputconfiguration-provider. (Viva dependency injection!!).
    Add following lines in your DSL's runtime-module-class' configure-method that is located in your DSL's main-folder:

      binder.bind(IOutputConfigurationProvider.class)
    .to(CustomOutputConfigurationProvider.class)
    .in(Singleton.class);

That should be it.

Tags : ,

Export this post as PDF document  Export this post to PDF document

Social Bookmarks :  Add this post to Slashdot    Add this post to Digg    Add this post to Reddit    Add this post to Delicious    Add this post to Stumble it    Add this post to Google    Add this post to Technorati    Add this post to Bloglines    Add this post to Facebook    Add this post to Furl    Add this post to Windows Live    Add this post to Yahoo!

Docker: Use host-network on docker container

If running a docker-container you have a network of your own per container. If you want to use the host-system's network instead you can use the --net=host option.

In this sample you call the docker containers arangodb arangoimp-call and connect with 127.0.0.1 which would not be possible if not using --net=host e.g.:

sudo docker run --net=host --rm --volumes-from arangodb-data_3.0 \
    -v $(pwd):/backup arangodb:3.0.4 /usr/bin/arangoimp \
    --server.endpoint tcp://127.0.0.1:8530 --server.database pwatch \
    --server.username uname --server.password pass
Tags : ,

Export this post as PDF document  Export this post to PDF document

Social Bookmarks :  Add this post to Slashdot    Add this post to Digg    Add this post to Reddit    Add this post to Delicious    Add this post to Stumble it    Add this post to Google    Add this post to Technorati    Add this post to Bloglines    Add this post to Facebook    Add this post to Furl    Add this post to Windows Live    Add this post to Yahoo!

Docker Bash

How to connect to a docker shell of an running docker-container

$ sudo docker exec -i -t 665b4a1e17b6 /bin/bash #by ID or $ sudo docker exec -i -t loving_heisenberg /bin/bash #by Name $ root@665b4a1e17b6:/#

If you want to get on the CLI of an running docker-container just use docker's exec command.

sudo docker

  • by-name:

  • First list all running containers:

    sudo docker ps

This will result in something like this: docker ps

Now you can call the container's bash-exe and you get on the CLI. You can do this by name or by container-id

  • by name

    sudo docker exec -i -t arangodb-server /bin/bash
  • by container-id

    sudo docker exec -i -t 8fd98d632eb2 /bin/bash
Tags : ,

Export this post as PDF document  Export this post to PDF document

Social Bookmarks :  Add this post to Slashdot    Add this post to Digg    Add this post to Reddit    Add this post to Delicious    Add this post to Stumble it    Add this post to Google    Add this post to Technorati    Add this post to Bloglines    Add this post to Facebook    Add this post to Furl    Add this post to Windows Live    Add this post to Yahoo!

Unban a client-ip in sshd after failing passwords

...if you have still somehow access to the server

What happend?
Today I was working on my synced test-server.
In order to check the apache2 is working properly I set the domain to test in my /etc/hosts file. Good thing first. Everything is working fine, just as if I would work on the production server.

But when you want to login to the production-server and you use the overridden url you end up again on the test-system (where I changed the password for safety reasons). To make the long story short. I ended up failing the password three times and when I realized that I was trying to access the test-server it was too late. The test-server banned me. At least for some time.

Good that I have the VM 'physically' here and can work directly on the test-server. So here is the magic how to tell fail2ban so unban a certain IP:

# check for the right jail name. (I guess this will be everytime sshd)
sudo fail2ban-client status
# unban
sudo fail2ban-client set <JAIL> unbanip <THE_IP_TO_UNBAN>
# e.g.  sudo fail2ban-client set sshd unbanip 192.168.0.95

Export this post as PDF document  Export this post to PDF document

Social Bookmarks :  Add this post to Slashdot    Add this post to Digg    Add this post to Reddit    Add this post to Delicious    Add this post to Stumble it    Add this post to Google    Add this post to Technorati    Add this post to Bloglines    Add this post to Facebook    Add this post to Furl    Add this post to Windows Live    Add this post to Yahoo!
<< Previous | Home | Next >>