Some code just kills your day...

Ignite at GeekOut 2013

On the 13th and 14th of June Ignite has been to GeekOut conference in Tallinn. This was a 2-day JVM conference, organized by ZeroTurnaround guys at Salme Kultuurikeskus.

Last years conference was a success, so this year Ignite delegation was huge, 6 people. Huge for a company of that size of course :)

GeekOut is one of a kind event in Estonia and number of participants grows each year. On the other hand, there is one important difference from conferences that happen in other countries - it’s local and local means that there are many familiar faces.

Conference was quite nicely organized. Networking was really brought to the front line. There was enough time between talks to meet colleagues from other companies as well as new interesting people. One of the best parts of the conference was networking event - party at F-hoone.

On Day-one Geert Bevin from ZeroTurnaround was entertaining the audience with his Eigenharp in a speech “Programmers are way cooler than musicians”. Java-world celebrity presenter Juergen Hoeller gave us insight on the future of Spring & Java in his “Spring 4 on Java 8” speech.

Second day will be remembered by hilarious presentation from Sven Peters (Atlassian) on the topic of “How To Do Kick-Ass Software Development”.

Beside these there were many other interesting talks.

Trade show organization was quite unusual. Vendors did not hide in the booth and took the stage during breaks between main talks.

Thanks to ZeroTurnaround guys for organizing such a great event!

Eclipse and WebKit on Ubuntu 64

Yes, Eclipse does crash sometimes, but we still love it, for some reason :)

If your crash report says something like:

# C  [libwebkitgtk-1.0.so.0+0x426915]  webkitWebViewRegisterForIconNotification+0xb5

then today is your lucky day. I have a solution for you.

First of all, about the problem

Looks like Eclipse does not play nice with libwebkitgtk-1.0, it wants libwebkitgtk-3. At least on Ubuntu 64. Most probably, you do have both of them installed, but for some reason Eclipse picks version 1 instead of version 3.

Solution

Well, first solution is quite obvious - remove bloody libwebkitgtk-1.0. Easy shortcut to happy place without any crashes. However, there is a small issue with that approach. If you do have  package installed, then probably there is a reason and you would have to remove some other packages that depend on libwebkitgtk-1.0. In my case, Gimp was the cornerstone. It does not work with libwebkitgtk-3.Too bad.

Second solution is a bit more sophisticated. It is the reason why I like Ubuntu and at the same time reason for many jokes and criticisms from other OS users. If something does not work the way we want - let's recompile it :D

If you take a look at WebKit source, you will notice that webkitWebViewRegisterForIconNotification is called only #if ENABLE(ICONDATABASE). What?! I'm getting crashes because of some icons?! I hardly care if I have those icons in my main browser, not speaking about browser integrated in Eclipse or Gimp. Let's kill icons!

I will not describe in details how to recompile the package, there are lot's of materials on that topic. When you get WebKit source, search Debian configuration files for keys that disable icon database. Unfortunately, there is trick. Looks like disabling of icon database was not tested by developers. There are some places in code that do not have correct compile instructions and will complain about missing methods. I suppose if you got to that point, then surrounding some lines of code with compiler instructions is not a problem for you :)

If it sounds too scary for you, here is my version of package.

Now you just have to install it: 

dpkg --install libwebkitgtk-1.0-0_1.10.0-0ubuntu1.1_amd64.deb

There is one issue with my package. I did not change version. When you install it, synaptic will show that there is an update available. Next time when you will install a bunch of updates - it will be replaced with the one from repository. Best solution is to use "lock version" in synaptic or something similar, but if custom package version is the same as version in repository - it will not help. This means that you would have to check the list of updates that you update manager is going to install and make sure that WebKit will not be updated. Ugly, but works, as a temporary solution. If you will compile own package - don't make my mistake - change version.

Using WiFi and network cable at the same time in Ubuntu

Did you ever have to switch between WiFi and network cable several times a day? If yes, than this blog post is for you.

DISCLAIMER: Check your security policy before trying out solutions from this blog post.

Actually, it does not have to be a WiFi and a network cable. It can be two WiFi connections or two network cable connections. The point is, that you have 2 (or more) network interfaces.

Usually, I have this situation when I need to use intranet resources, which are available for local network and this network has "issues" with connecting to outside world.

By default, Ubuntu allows you to have multiple network connections at the same time. The problem is, that it needs some help on deciding which one to use in certain circumstances.

On this screenshot you can see two active connections:

"Wired connection 1" - default name for wired network connections in Ubuntu. Let's assume that this is a local network that does not have access to internet.

"aleksz-wifi" - wireless connection. Has access to internet resources, like Skype.

How does Ubuntu know which one of those connections should be used to access public web? Simple, there is a routing table. You can see it with "route" command.

The Routing Table

That's what I get. I did not configure anything yet. You can assume this routing table to be more or less random.

Probably, you would not be reading this post if you would understand how routing table works, so I will try to explain it a bit. In last column ("Iface"), you see interface name. eth0 is for "Wired connection 1" network and wlan0 is for "aleksz-wifi". First column ("Destination") together with third column ("Genmask") form the range of IP addresses for which this interface should be used. That's actually what we need. By changing routing table, we can say that one range of IP addresses should use one interface, and another range should use the other interface.

In given scenario, we need two ranges. On screenshot, you might have noticed that there is a "default" destination. This means, that we actually have to define rule for one range and all other addresses will be using "default" rule.

Default interface

Now you should think what interface will be "default". In given scenario, there are two facts that make us choose wlan0 as a default interface:

1. We are going to use Skype on that interface. As Skype uses P2P network, it will be quite hard to identify destination IP range.
2. If routing table hits some unknown IP, then most probably it will be an external resource.

If you pay attention, then you will notice that routing table on screenshot puts eth0 as default interface.

Changing configuration

There are many ways to  configure routing table, but we are going to use the easiest one.

Open "Edit Connections..." dialog

Open "Wired" tab and click "Add" button. We are going to create a new configuration that can be enabled manually 

Enter some "Connection name" and choose "Device MAC address" . Dropdown should contain MAC for eth0 interface. Uncheck "Connect automatically" check box if this is not your primary setting for wired connections.

Next go to "IPv4 Settings" tab (or v6, depending on your network). Click "Routes..." button.

In case if you are going to use this connection to access only resources available on local subnet - check "Use this connection only for resources on its network". Save your new configuration.

Now you can choose "my intranet" for wired connection

Let's check the routing table:

Here you go, default interface is wlan0. eth0 is still used for local connections.

That was easy. However, in real world it is highly possible that your local network structure will not be as plain as in my example. You might need access to resources on other subnets. This requires a bit more complex routing configuration:

Except for local subnet addresses, this configuration resolves addresses in other subnets through gateway.

Google Calendar API

Why do we care about Google Calendar API?

Reuse

Development is expensive. Let's face it. Hacking through all kind of development issues takes time and time costs, quite a lot. One of the solutions to this problem is reusing.

Developers reuse all the time - libraries, application servers, even "Googling" for errors. Using remote API is quite the same, but in larger scale.

Integrate

Using publicly available service API has one important feature that makes it different from library - integration. Probably you are not the only one who integrates with that service. You join the community.

Nowadays, most popular web resources offer remote APIs. Google is one of them. It offers wide range of services that people use in everyday life, covering many trivial and not so trivial tasks that might be required in your project. And guess what? Google wants to share it with you! For this purpose they offer impressive amount of different APIs. One of them is Google Calendar API and today we are going to focus on that. If you are not that interested in calendar, this blog still might give you some hints about other APIs, because there are many things in common.

What can you do with it?

Once upon a time we had a research project. It was not about remote APIs. It was about using HTML5 for mobile development. At the same time, we wanted to make something useful. That's how Conference App was born.

Idea behind this application is to make a convenient conference schedule for mobile devices. Idea comes from personal experience, as many people from our company visit conferences and some even organize them. Usually, schedule printed on paper is either informative or compact, but not both at the same time.

Some time ago I used Google calendar for storing conference schedule and it worked. I was able to enter all events on my laptop, I was able to check the schedule on my Android device and I was able to share this calendar. However, it was a bit inconvenient. Mainly because of the way how many events happening at the same time are displayed.

Our application was supposed to take all the good stuff, fix the bad stuff and put it into conference context. Google Calendar desktop UI can handle lot's of complex stuff, but it comes with a price. We removed all the complexity and left only what is need for current task.

This is a screenshot from conference app. It shows exactly the same calendar.

How does it work?

Now we are done with introduction and will move to stuff that should interest developers a bit more. Let's see some technical details.

REST

Google Calendar API 3 uses REST. Payload is JSON. It replaced Atom based GData protocol. Old version 2 API is still there, but I'm not sure if it will be discarded at some point. Actually, initially our Conference App was using version 2 and I migrated it to version 3 only when started preparing this blog post. Migration to new version is not very tricky. There is a nice migration guide. If you are still using API v2 - migrate.

Client libraries

REST API assumes that you can make requests to it using any HTTP capable tool and format of requests and responses is described in details in reference documentation. However, Google APIs are quite complex and it is highly recommended to use client libraries. Client libraries are available for quite a large amount of languages and frameworks.

Python is in the center, because this is the only library that does not have "alpha" or "beta" status. Libraries above Python have "beta" status and libraries below have "alpha" status. We used Java library and did not have any significant issues, even though it has "beta" status.

Collections

As any other well structured REST API, Google Calendar API is split into collections of resources.

• ACL
• CalendarList
• Calendars
• Colors
• Events
• Freebusy
• Settings

Most of those collections support CRUD operations. Some support only a subset of CRUD. Some have extra operations. This is a typical set:

• delete
• get
• list
• insert
• update
• patch
• ...

Putting it all together

This is a Java sample that takes event summary from each item in "Events" collection. Parameter to "list" method is not a real calendar id, so sample will not work.

Events events = calendar
 .events()
 .list("eimems5@group.calendar.google.com")
 .execute();

for (Event event : events.getItems()) {
event.getSummary();
}

Quite easy. If someone has seen second version of this API, he will notice, that now it takes much less code.

Authorization

It's quite obvious, that remote API access has to be secure. All requests to Google Calendar API require OAuth2 authorization. Actually, not only Calendar API. Other APIs use same process. 

Idea behind OAuth2 authorization is quite simple and elegant, but implementation is not always that obvious and it is another reason for using client libraries.

By the way, there is a simpler OAuth version 1 that you can use, but version 2 is preferred.

Different applications (and even different devices) require different authorization flows and Google has something to offer.

Client-side applications

JavaScript-centric applications. These applications may access a Google API while the user is present at the application.

Web server applications

These applications may access a Google API while the user is present at the application or after the user has left the application. This type of access to a Google API is called offline, since the user does not have to be present at the browser.

Installed application

These applications are distributed to individual machines. These applications may access a Google API while the user is present at the application or when the application is running in the background for long periods of time without direct interaction with the user. Like email checking in Android device.

Device applications

Applications that run on devices with limited input capabilities (e.g. game consoles, video cameras, printers) may access an API on behalf of a user, but the user must have separate access to a computer or device with richer input capabilities.

Service accounts

Service accounts can be used on behalf of an application when it does not access user information or accesses publicly available information. Used to identify application. Our application uses this scenario.

Scopes

Each authorization request provides scope that it want's to access. In other words, application might have a permission to read your calendar events, but without modification option.

Google will ask the user if he agrees with requested permissions.

Limits

Almost all Google APIs have limits, which is quite logical. For example, Calendar API has a limit of 10000 requests/day. In some cases it is possible to exceed the limits for extra fee according to the price list. In case of Calendar API there is no price list, only a form that you can fill to request quota changes. Not sure how it works, never tried. 10000 requests per day was more than enough for our project due to long time caching and the fact that we are using only one calendar. Basically this means that we are making only about 144 requests per day.

In case if your app is accessing user calendars, then amount of requests will depend on the amount of users. For this case, there is another limit that you can set in Google API Console - requests/second/user. Default is 5, but you can set it yourself. Personally, I think that you should use this feature only as a last resort and control it in application.

Other services

As I said before, Google offers APIs for quite a large range of services. Here are some of them. Don't be confused with font sizes in this cloud. Bigger font means my personal interest. Hope you can find something interesting for yourself!

Eclipse hint

I'm sure you had situations when Eclipse was behaving strange. Probably your first attempt is to clean and refresh the project. Sometimes it's not enough, but luckily there is a "brute force" method:

1) Delete project from workspace (don't delete it from filesystem)

2) Delete .settings, .project and .classpath and target (assuming that you use Maven)

3) Import project into workspace

There you go. Fresh checkout is another option, but does not work very well if you have uncommitted changes.

Continuous delivery with Jenkins and Gradle

  To my mind, continuous delivery is a great thing that gives real value. However, it seems that setting up a proper build pipeline to support it is not a trivial task.

  My first attempt to make a build pipeline was based on Maven. I was using FOSS Jenkins instance, provided by CloudBees.

  Pipeline steps that interested me were:

1. snapshot - runs unit tests and creates build artifact (WAR in my case)
2. REST integration tests - deploys WAR and tests it's REST interfaces
3. UI tests - deploys WAR and tests it's UI in different browsers
4. release - updates versions and puts tags in SCM
5. ... endless amount of other pipeline tasks

Some other requirements were:

1. Use same SCM code version for all steps in pipeline
2. Use only those Jenkins plugins which are available for free on CloudBees

  It was pretty easy to compile all required tasks in Maven life cycle and run them in sequence. However, as soon as I've started configuring jobs in Jenkins, it became clear that Maven life cycle does not match my pipeline. Simple "mvn install" turned into "mvn clean deploy -DskipITs" and "mvn failsafe:integration-test -P selenium". It was especially hard to configure jobs in the middle on the pipeline, because Maven tends to execute all preceding life cycle phases.

  You can find some of my attempts here: https://reference.ci.cloudbees.com/. In the end, I got it working but it did not feel right for sure.

  On June 14th I've visited conference called GeekOut, where Hans Dockter was showing Gradle. At first it seemed pretty much like Maven, but it had some features that looked quite promising to me:

1. Gradle plugins provide Maven simplicity
2. At the same time, it's incredibly easy to customize life cycle
3. Incremental builds

  Here are my attempts to make a pipeline with Gradle: https://reference.ci.cloudbees.com/view/gradle/. Thanks to flexibility of Gradle I was able to fix life cycle according to my needs and incremental builds took care of skipping preceding steps. "mvn failsafe:integration-test -P selenium" becomes "gradle cleanUiTest uiTest". Now it does feel right!

  Some hints:

1. If you want to start incremental build in one Jenkins job and resume it in other - make sure that absolute path to workspace stays the same. Otherwise, Gradle will start build all over. For example, if first job uses workspace /scratch/jenkins/workspace/1-snapshot-gradle and the second one uses /scratch/jenkins/workspace/2-rest-test-gradle, second job will not resume incremental build.