http/tls connection in python, android, EC2

Have a side project to build a standard tls package for the team. Ive never tried socket, so start with python just to get a feel. Following is my own experiment, to connect the server side code on EC2, and the client side code on my local laptop.

1. simple http
I use sample code from official doc. It is really simple. All you need to do other than code is configure the port for EC2 instance.

For the instance you are running, configure its security group so that the specific port you want to communicate on is open like 2727 above.

2. simple https
Things get rough with security. Basically, what I know about https, i.e. tls, is that it utilizes a public key identification system to secure the communication via http. The server has a private key, which is only known to itself. It also has a corresponding public key ready to distribute to anyone need to communicate with it. In order for the other side to trust it, the server has to have its public key certified by trusted 3rd party, called a Certificate Authority. Same with client side.

However, if we just want a connection between our own server and client, we could generate keys and certificates ourselves without paying for CA cert file. This is called self-signed certificate, or root CA certificate.

openssl req -new -x509 -days 365 -nodes -out cert.pem -keyout cert.pem

If you have openssl installed on your computer, you could use it to generate keys. In this case, I generate private key and certificate in the same file. Then I just copy it to the other side. Both sides use the same keys. Things are simpler here, for which most cases you might wanna use a more secure authority to certify for you.

Then both sides I use sample scripts from official doc again. Note for https connection you also have to open the port for ec2 instance.

3. simple https with android
With android things are bit complex with certificates. I have this cert.pem file, which is not enough for android. Bouncy Castle encryption is supported well by android, which is the one we are gonna use to generate client side key file.

First is to install Bouncy Castle. Note android is using a different version of it, version 145, not 146 from official site. Find one, download it, a jar file. Put it in the directory '/usr/libexec/java_home/lib/ext', where on mac should be '/System/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Home/lib/ext'. Second, add following sentence into the jave.security file also located in lib folder:


Having keytool in your machine, do following with the cert.pem file:
Now you will have mykeystore.bks file in raw directory. I here use a der file because android returns 'wrong version of certificate' error. To generate der file from pem:


We are almost done here. Just grab any sample code for https connection in android, using whether httpURLconnection or httpclient, put correct password and file name into place, everything should be fine now.

Project update: location

In my project, the mobile application is supposed to periodically obtain user's current location and record it into a database; this process should be done even when user quits the app. Previously, my solution is to set a recurring task using AlarmManager, which will send an alarm every 30 seconds after user starts it; another class, extends AlarmReceiver, will start a service every time it receives the alarm task; lastly, in the service class, I create a location manager to obtain location using getLastKnownLocation().

This implementation, technically, issues location tracking task exact every 30 seconds. Two problems with this. First is, although the alarm is generated every 30 seconds, the location tracking might not be finished within 30 seconds every time. There might a situation that the mobile phone has not got any location for the current alarm task when the next one is generated. I am not so sure if this is 100% correct, but I do find sometimes there will be delay in updating location, like no location update for 5 minutes, then several new location updates appear in database with the same coordinates. Anyway, I am suspecting it is because my Activity Alarm -> Service -> LocationManager process take too much time. Another problem is, every task the mobile phone creates a new service, which seems inefficient and unnecessary.

Several days ago I find this post writing about LocationManager and LocationListener, come to my rescue. Basically it explains how the method requestLocationUpdates(long minTime, float minDistance, Criteria criteria, PendingIntent intent) works. This methods aims at saving energy while giving accurate location tracking. The minTime parameter gives the time duration that the location provider will rest (status changes into unavailable) before it activates and obtain location again. The actual time interval will be equal or greater than minTime, due to several reasons: location update only be sent to the app if the difference between new and old location is greater than minDistance, also the provider might take some time to obtain the latest location. This definitely takes longer time than getLastKnownLocation(), since the latter one uses cached location.

Anyway, it seems I could use this request method directly for my periodically update, while it is not exact update, it has several advantages: there will only updates if necessary, if user does not move, my previous version will still update as alarms go on and off; also it is definitely more efficient than multiple services. Now my activity will start the service after user presses the start button, which will then register a LocationListener and request location update. Note this request will not stop until the service stops or I explicitly use removeUpdate method. Lastly, in my class of Location Listener, I put location into database in the onLocationChanged method: it gets invoked every time there is a location update from the location listener.

Some digress, I just find this Notification class which could perfectly display the app status in the notification bar. Previously, I use a textview to hold the status passed from service, but setting it globally visible is better, especially when user quits the app.


I use the sample code comes with SDK. But it is somehow deprecated, thus I change a little. This snippet of code would be put in the onCreate method of my service, so that every time my service starts there will be a notification in the bar. I also put a cancel method in the onDestroy method so that it gets killed when the service stops. Note Notification class is in API level 11 (Honeycomb), which means if you are using the same API level I use (2.3.3, level 10, GingerBread) or lower, you need to import support package in your project.

I will test this new version tomorrow.

Toggle Button and Alert Dialog

Actually I tried them before and now I dont need them in my project. But write down them just for reference.

Both of them are ways to switch among given states; toggle button has 2 states, "on" and "off", while Alert Dialog could support unlimited ones, technically. I tried them for yes/no switch because I want to let user choose whether they want only GPS (fine location tracker) or currently best one to track the location in my project.

Toggle Button is really just switching the text on the button actually. At the same time, you could extract its text at other places, like another button. User could switch to "only GPS" or "all providers" before they hit the tracking button. In the tracking button, I will first extract the text of the toggle button, and then put different info in intent to start the tracking service.

To detect which state user chooses, I create two strings in the string.xml to hold the text for two states, and then compare the current text with them. Nothing fancy here; easy to go.

Alert Dialog is another class to let user choose options. When enabled, the app will pop up a window containing options. In current version of Android, we could use AlertDialog.Builder class to build the alert dialog with the title, content, on click methods we need, and then create it.

The doPositiveClick and doNegativeClick are two methods corresponding to two choices. What they do in my version is just passing different int to the service class in the intent; the service will initiate different location listener based on the int.

Android: Broadcast locally within the app

Previously, when I need to pass some info between my Activity and Service classes, I will use Broadcast class to send a new intent containing the info from the service side, then register a receiver at the activity side. However, Broadcast class is mainly used to send info across apps, e.g. you could send some info from your app to the calendar app to create a new calendar event: it is not for communication occurred inside one app. Exposing your in-app info globally might raise security issues and also not efficient.

OK, then I find this LocalBroadcastManager class, which, according to its name, mange the local broadcast, i.e. passing info within the app (from one app component to another). This is perfect for my use. While the official doc does not provide any details on how to use it, here is an excellent tutorial covering everything we need. I summarize below just for reference:

1. get the library
This class belongs to the support package, which means you have to add the package as a 3rd-party lib  and then import android.support.v4.content.LocalBroadcastManager.

2. create the sender
Pretty straightforward.

3. create the receiver
To receive any intents, globally or locally, you first have to create and register your own broadcast receiver:

All done. Very convenient to use.

Lab project update 3: asyncTask, db file export

This is a minor update, to add the function of exporting local db file onto SD card. The reason why we do export instead of directly dragging the db file using some file explorer app it because, when testing on device, unless you root your phone, you will not be able to access your db files using apps.

Looking through android doc (btw, newest official site looks absolutely sexy) and stack overflow, I decide to use AsyncTask class to do this background job. It looks like Service, but is easier to use to communicate with the main UI thread. Unlike Service class, in which you have to care about when to create, start, handle message, stop the thread, this class provides exact functions that wrap up those details, including one before you start the task (onPreExecute), one to do background job (doInBackground), one to update main thread if you want to (onProgressUpdate) and one to return some results after the task is finished (onPostExecute). Details could be found in the doc.

Normally, it requires to override at least the doInBackground method, also in most of the time the onPostExecute method. My second method is nothing new, just making a Toast to indicate whether file has been correctly exported. My first method:

Let's go through each step. Line 3 is to locate the db file you want to export. I use the Environment class to obtain path info. Note in android, the db file of an app is created in the path "/data/data/your.package.name/databases/your_db_name.db".  The method getDataDirectory() will return the first "/data" therefore for LOCTABLE_PATH you only need to add the path after it. Line 5 gets the external dic state, which I use in line 6 to detect if the SD card is writable, defined as MEDIA_MOUNTED. If SD card is not available, I will just return a Toast and finish the task.

Line 7 calls the getExternalStorageDirectory() to obtain the SD card dir, which should be "/mnt/sdcard", you could define the dir you would like to save your db file as EXPORT_PATH. The following if statement is to check if the path you want already exists, otherwise create it. Line 11 is to create a file object at your given path. Note currently you haven't created an actual file, you just create an object and make it ready to generate a file. Also, in order to write to the external disc, you have to add following permission in your AndroidManifest.xml:

Lastly the try block is to create the file and copy your db file to it. The copyfile method should be defined by you according to what kind of file you want. I recommend just copying raw content into a .db file and then open/read it in a db browser like this. Raw file copying method in java could be found here.

Using Google places API in Android

Google places API is a service provides information about places. Basically, it will take coordinates and other additional parameters (like city, limits, etc) as input, return information related to the input such as name, address, types of nearby places. Think about Google map and their street-view car; the accuracy of this API should not be bad. I take this as an alternative to the Foursquare API to obtain places information. Actually, I expect it outperforms 4sq because most of places in 4sq are created by users, which leads to inevitable noise.

Anyway, I look into the java library of places API since I need them in my android app. Turns out the bad news is they don't have any specific java lib; moreover, 3rd party java libs I find on github are poorly supported. The good news is, Google provides a general "Google API console" and also its java lib, google-api-java-client. Although this lib does not support places api specifically, build a wrapper using this and places api is sufficient for me. The more good news is, I find this awesome blog and its corresponding sample on github. However the blog is written one year ago thus some methods it uses are deprecated in today's new version. I modified the sample to make it compatible with the latest java client, find it on my github.

Since the blog actually tells us everything we need to know about Places API, I will just skip the basics and write down how the process works in case I forget. Generally, to build your own application using this API, you need to always look into three web pages: the blog, the java doc of Google API and the official doc of places API. 

Basically there are two steps: to request places information using Places API and selective parameters, then parse the output into strings we need. First step, use HttpFactory as parameter to generate a Http request; the HttpFactory object contains a json parser and an arbitrary header. Then we use request.getUrl() to put every option we want to customize our request and send it using request.execute(). Now the second step will automatically be executed when the server returns the result. The result is in json, the json parser in httpFactory would parse it into a java model. Then the blog propose a very clever method (at least to me, a java newbie): create some classes to catch some specific strings from models. With the decorator "@Key" we could define what part of that result we want, e.g. name, types, etc. This is really efficient. Then you do whatever you want with the result.

Places API provides three search URL (I only look into search part), general search (return a list of nearby places), search detail (return the detailed information of a place), search autocomplete (predict and return places based on input). I only need the first two, but the third one is really cool: adding appropriate processing blocks, it could become a real-time prediction search just like google instant search.

Finally, there is one thing holds me back several times. The AndroidManifest.xml file. NOW REMEMBER: you have to explicitly indicate them if you add following components to your project:

1. permissions, internet, gps, etc;
2. service, including intent services;
3. content providers

Every time I use these components I forget to add them and then stuck with it for a while. Now it is done here and I will never make the same mistake again. Also, any networking thing (http GET/POST, for example) is not allowed to be done in the main thread; you have to use a service or similar technique.