Trafikanten & open data update - things are happening!

I am just back from a meeting with Trafikanten (where I met the Advisor for realtime data and the QA responsible). The meeting has been from my point of view pretty constructive. We've then spent a bit of time to better understand our needs. We've exchanged thoughts around their current data offering, the licensing, etc and came up with ideas that I think will make many people happy. I won't say much more for the moment, as I am waiting for them to actually implement the ideas we talked about, but as soon as I know more, you'll be posted. I would like again to thank Trafikanten for their time and I really wish them good luck with their changes. Exiting times!

Trafikanten: meeting on license and open data next week

A follow up after my last post on the topic of opendata/trafikanten. It seems that being constructive and open to new possibilities has some results. Trafikanten is revising their API license, in oart to be attractive to smaller actors as well. They invited me to a meeting on the 20th of January to present and discuss their new license proposal.

If you have some meaning on the topic, feel free to contact me, publicly or privately.

Publishing your signed OSS project to the Sonatype OSS repository

One thing I wanted to do was to publish the yui-runner releases to ibiblio. It's now pretty easy thanks to the Sonatype OSS repository, which will help you stage and release your repositories.

See the official Sonatype OSS Maven Repository guide.

Because of a few choices (git, scala), I ran into a few troubles or quirks that are worth noting though. Here's the outline of the procedure I picked up:

• clean up your POM as per the documentation
  
• no need yet to add sections for javadoc, pgp, release etc. They are by default included once the parent is inherited from. If you wish to test javadoc or pgp, you should then enable the sonatype-oss-release profile to your maven execution when needed.
  
• if you do add the javadoc attach-sources goal, you might ran into problems that causes the reactor to not find the project's own internal dependencies. This is because I think that particular goal forks maven. See https://github.com/lacostej/yui-runner/commit/7f15baa5a73c0da773a283800521a12d8c20a905
  
• you might want to configure the git push off by default (http://jira.codehaus.org/browse/SCM-444). This I think fits well with the staging principle of the Sonatype OSS repo. Use something like:
  

  
       <plugin>
         <groupId>org.apache.maven.plugins</groupId>
         <artifactId>maven-release-plugin</artifactId>
         <version>2.1</version>
         <configuration>
           <pushChanges>false</pushChanges>
         </configuration>
       </plugin>
  

  
  
• during release, you might want to add the use of the gpg agent if you don't like to enter your GPG passphrase for every module
  

  
       <plugin>
         <groupId>org.apache.maven.plugins</groupId>
         <artifactId>maven-gpg-plugin</artifactId>
         <version>1.1</version>
         <configuration>
           <useAgent>true</useAgent>
         </configuration>
       </plugin>
  

  
  
• the Sonatype OSS repository expects releases to have a javadoc artifact. As the maven-scala-plugin doesn't yet behave like the maven-javadoc-plugin, and doesn't produce nor attach a javadoc artifact to your build, I cheated a bit. This is of course temporary.
  
  
• Deploying will require the use of GPG signatures. This is what i did on Ubuntu 10.10 to get it working as I wanted (with a gpg-agent):
  
  
  
  1. create a signing key:
     

     
     jerome@lenovo-w510:~ $ gpg --gen-key
     jerome@lenovo-w510:~ $ gpg --list-keys
     jerome@lenovo-w510:~ $ gpg --keyserver pgp.mit.edu --send-key 576831db3ae26aaf
     

     
     
  2. install the agent:
     

     
     root@lenovo-w510:~ # apt-get install gnupg-agent
     

     
     
  3. configure the gpg-agent. Under ~/.gnupg/gpg-agent.conf, write
     

     
     pinentry-program /usr/bin/pinentry-gtk-2
     default-cache-ttl 86400
     max-cache-ttl 86400
     

     
     
  
  
  4. start the agent and copy the output in the terminal (this can be moved to your DE session startup instead later on)
     

     
     jerome@lenovo-w510:~ $ gpg-agent --daemon
     GPG_AGENT_INFO=/tmp/gpg-t61OEn/S.gpg-agent:28820:1; export GPG_AGENT_INFO;
     jerome@lenovo-w510:~ $ GPG_AGENT_INFO=/tmp/gpg-t61OEn/S.gpg-agent:28820:1; export GPG_AGENT_INFO;
     

     
     
  
  
• Now the release takes the form of:
  
  
  
  1. clean your directory (git status, git ignore, etc)
     
  2. mvn release:clean (optional)
     
  3. mvn release:prepare
     
  4. git push --tags && git push (if you disabled push)
     
  5. mvn release:perform
     
  6. follow the Sonatype doc for testing your staging repo and releasing it. If you run into problems, don't forget to drop your repo before releasing again.
     
  
  

yui-runner, what it's for ?

I've released the first public version of yui-runner, a small framework for automating the running of YUI javascript unit tests and reporting of their results.

See https://github.com/lacostej/yui-runner

As the main goal is to be able to run your tests from anywhere, the integration is done thanks to a JUnit runner. This means your tests will run in maven thanks to surefire or in your favorite editor.

An example project can be found under https://github.com/lacostej/yui-runner/tree/master/yui-runner-it/src/test/resources.

The project will soon be available on ibiblio, something I'll blog about.

Note: the API for yui-runner is going to change for more flexibility in selecting the URLs to load.

user.js scripts for browser (firefox & chromium)

I wrote a Greasemonkey script for a friend: Disable keyboard input. As strange as it sounds, the goal is to let someone navigate with a mouse but not use they keyboard to enter stuff. The user suffers Alzheimer's disease.

I found out by accident that chrome also supports firefox greasemonkey user.js. It treats them as extensions. Now chromium doesn't seem to have a GUI to manage the installed scripts, and as it changes the script's file name upon installation, I had to use grep to find the proper file. On linux, that particular installation went under:

/home/jerome/.config/chromium/Default/Extensions/hlikiangigedihjlmhceanglikpccjoi/1.0_0/script.js

Stop the browser, delete the directory and restart the browser to get the extension removed.

Intellij issues on Ubuntu 64 bits with encrypted /home

Kept running into strange index issues with my Intellij 9 and 10 installations. The system/log/idea.log file was filled with:

java.lang.RuntimeException: java.lang.IllegalArgumentException: can't position buffer to offset 43716
        at com.intellij.util.io.PersistentEnumerator.valueOf(PersistentEnumerator.java:499)
        at com.intellij.util.io.StringRef.getString(StringRef.java:48)
        at com.intellij.util.io.StringRef.toString(StringRef.java:92)
[..]
Caused by: java.lang.IllegalArgumentException: can't position buffer to offset 43716
        at com.intellij.util.io.PagedFileStorage.get(PagedFileStorage.java:181)
        at com.intellij.util.io.PagedFileStorage.getInt(PagedFileStorage.java:127)
        at com.intellij.util.io.ResizeableMappedFile.getInt(ResizeableMappedFile.java:141)
[..]

or

java.lang.IndexOutOfBoundsException: off=1006632960 key.owner.length()=17582
        at com.intellij.util.io.PagedFileStorage$StorageLock$BuffersCache.createValue(PagedFileStorage.java:66)
        at com.intellij.util.io.PagedFileStorage$MyCache.get(PagedFileStorage.java:329)
        at com.intellij.util.io.PagedFileStorage.getBuffer(PagedFileStorage.java:280)
        at com.intellij.util.io.PagedFileStorage.get(PagedFileStorage.java:176)
        at com.intellij.util.io.PagedFileStorage.getInt(PagedFileStorage.java:127)
        at com.intellij.util.io.ResizeableMappedFile.getInt(ResizeableMappedFile.java:141)
        at com.intellij.util.io.PersistentEnumerator.hashCodeOf(PersistentEnumerator.java:479)
        at com.intellij.util.io.PersistentEnumerator.enumerateImpl(PersistentEnumerator.java:328)
        at com.intellij.util.io.PersistentEnumerator.enumerate(PersistentEnumerator.java:222)

I seem to have fixed the issue by moving the IDEA config and caches into a non encrypted partition.

$ diff -u /usr/local/lib/idea/bin/idea.properties.orig /usr/local/lib/idea/bin/idea.properties
--- /usr/local/lib/idea/bin/idea.properties.orig 2010-12-10 15:50:01.916136005 +0100
+++ /usr/local/lib/idea/bin/idea.properties 2010-12-10 15:50:44.096136000 +0100
@@ -10,13 +10,13 @@
 # Note for Windows users: please make sure you're using forward slashes. I.e. c:/idea/system
 
 # path to IDEA config folder. Make sure you're using forward slashes
-idea.config.path=${user.home}/.IntelliJIdea10/config
+idea.config.path=/usr/local/${user.home}/.IntelliJIdea10/config
 
 # path to IDEA system folder. Make sure you're using forward slashes
-idea.system.path=${user.home}/.IntelliJIdea10/system
+idea.system.path=/usr/local/${user.home}/.IntelliJIdea10/system
 
 # path to user installed plugins folder. Make sure you're using forward slashes
-idea.plugins.path=${user.home}/.IntelliJIdea10/config/plugins
+idea.plugins.path=/usr/local/${user.home}/.IntelliJIdea10/config/plugins
 
 #---------------------------------------------------------------------
 # Maximum file size (kilobytes) IDEA should provide code assistance for.

See also http://devnet.jetbrains.net/docs/DOC-181 and http://devnet.jetbrains.net/message/5259840.

Git svn tree corruption

I use git over svn for all my subversion projects. This morning I hit the following problem:


$ git svn rebase
error: garbage at end of loose object 'd0df538270dcaf65a232fb4127b7c59212420129'
fatal: object d0df538270dcaf65a232fb4127b7c59212420129 is corrupted
diff-index HEAD --: command returned error: 128


Mmm no backup accessible right now, and no way I am pulling remotely a git svn for this 7 years old tree.

First let's find information on the commit.

$ mv .git/objects/d0/df538270dcaf65a232fb4127b7c59212420129 ../d0df538270dcaf65a232fb4127b7c59212420129
$ git fsck --full
broken link from tree 066ce753000ca9e1cfbd07a709cac4c7c3e5da47
to tree d0df538270dcaf65a232fb4127b7c59212420129
dangling tree 866d5ac672d6ef4009319e176f195fd01697165c
missing tree d0df538270dcaf65a232fb4127b7c59212420129


What does the previous tree contains ?

$ git ls-tree 066ce753000ca9e1cfbd07a709cac4c7c3e5da47
100755 blob 888ed870384e87dad0642199b4d7d938246946a8 xxx1
100755 blob 346bd13c9ba0d15046af891ccc88aa225d3c7ee3 xxx2
100755 blob 2eef3f1f2985ff1d9cdd4a23c8ebca54a9af1010 xxx3
100644 blob 6a501e34ee0fb197fa6770b2a8cc40cdf276607a xxx4
100755 blob fe61ef5d88201a7502057a2ff5fdd45db44f5075 xxx5
040000 tree 38354b54d3dca537d9c7c9d9ec1668c0c7287bc9 ddd1
040000 tree 50be3bfee5b37fc97b379ae04fe961a212a70a32 ddd2
040000 tree 6b108f1b534bb9465c2e31efe51d06fb05499376 ddd3
040000 tree 532a6da6ece3151b43a1cd0fd174fdadb071e402 ddd4
040000 tree 45363718750ed5142fad1d7be311581748057297 ddd5
040000 tree d0df538270dcaf65a232fb4127b7c59212420129 ddd6
040000 tree b77d3e9240185ed148727e01d4d03ad726411724 ddd7
040000 tree 8b69621e177132f3dc822c427501f1921f39bbe1 ddd8


Didn't help much.

In that SVN tree, my history is pretty linear. I try to find the commit before or after the one that fails:

$ git log --raw | grep svn+ssh | tail -1 | cut -d '@' -f 2
fatal: unable to read destination tree (d0df538270dcaf65a232fb4127b7c59212420129)
4321 20108eba-4dfa-0310-bbc5-8f3fd2a08cff


Log fails to find the full history, and I now know which commits maps to the broken tree: 4320: the one before the last succesffuly logged commit (4321).

Let's verify:

$ git svn find-rev r4320
4e0264e69a5289c5ac5c461085c900009c90c5b9
$ git show 4e0264e69a5289c5ac5c461085c900009c90c5b9
fatal: unable to read destination tree (d0df538270dcaf65a232fb4127b7c59212420129)


Now let's just fetch some versions around that commit and replace our broken tree file.


$ svn clone svn+ssh://blablabla.com/opt/svn-repository/xxx/trunk/xxx -r 4318:4322 xxx2.git
$ mv xxx2.git/.git/objects/d0/df538270dcaf65a232fb4127b7c59212420129 .git/objects/d0/


Check the commit ?

$ git show 4e0264e69a5289c5ac5c461085c900009c90c5b9


git svn rebase now works !