Django auth pt1

Nowadays, to use google api correctly in your web applications, most likely you have to use its own authentication oauth 2.0. They say it is simple and easy; to a newbie of web dev it is not. Anyway, I failed to integrate it with my web app, thusly I decide to learn auth from the scratch, and hope one day my web app will be popular enough to demand an authentication system (I mean it!).

Django has its own built-in auth package, which is ideal to use learn the concepts. Most importantly, built-in packages like this always come with built-in views, which could save you a lot of time to get started.

Let's see how to use built-in views: really simple.

In urls.py, we define three extra patterns besides admin ones. One is for the index view, the home page, which will require login authentication to see its content. The other two utilize the built-in views for login and logout. Note for login view you have to define a template to make it go if you don't want to copy the default template from django package; for logout I here use the logout_then_login view, which is pretty self-explained. Since it will redirect to login page automatically, we don't have to add any template for this view.

Now take a look at view.py:

Two things to know: @login_required decorator indicates that this view needs authentication, if currently the user is not logged in, it will take the user to the login view we defined above; if already logged in, it will render the index view as required.

I don't include the template it uses because you could find it in django doc. Only one thing is a little bit tricky: the default login view will pass several variables to the template, including one called next. This contains the url that the app will redirect to after successful login, which should be the url triggered this login (in our case it is "/" for index view). To make the redirect happens, in the template you have to define a redirect to the next, like this if your login page uses a form: <input type="hidden" name="next" value="{{ next }}"/>.

Lastly, remember to add a link to logout url in your index page in order to complete the whole login/logout process.

All done.

GRE helper project update 1

UPDATE: I cut this project. It turns out giving her a list generated by python script is much faster. Not everything needs a web app.

I start yet another django project on heroku today. It's a rather simple one, just to help my girlfriend memorize vocabulary and prepare her GRE test.

GRE is difficult for people whose first language is not English. The test involves so many non-daily English. You have to memorize, like hardcoded in your brain, thousands of words in order to just understand what this test is talking about, not mention other requirements. A typical preparation for GRE involves at least 4 months of memorizing words repeatedly. I have been through this before, it's a very stressful and boring period of time, especially when you spend several weeks and look back, realize you don't remember one damn word. Sadly I could not turn it into a happy thing; however, I think I could make it efficient and maybe shorten its duration somehow.

What I want it to do is let you type in how many vocabulary lists (do you want to know how many words Chinese have to memorize for just a single test?), then it will generate a calendar filled with each list as task, and distributed based on a predefined memory curve. Basically a customizable repetitive task generator hooked up with google calendar. I know it is still dull, but an organized and well-executed repetition on memorizing could efficiently reduce the amount of time you need to achieve your goal. For the calendar, I don't want to reinvent such a good calendar system, besides Google calendar API has a good python support.

Start with the model. It is pretty simple and common, but I intend to add maybe a list as a instance variable so that every list could have a list of dates on which they needs to be memorized. For the progress field, which is supposed to be within 0 to 100, I add two validators from django.core.validators to prevent overflow.


I make just one page, in which you will have a input area for you to type in how many lists you have, and a display area to show the calendar. The input with the submit button will issue a http POST request to server side, along with some parameters, including the number of your lists. Note django itself disable post method automatically to prevent CSRF attack. To reenable it:

1. add 'django.middleware.csrf.CsrfViewMiddleware' to the MIDDLEWARE_CLASSES in your settings.py (this should be done by default in the latest version now)
2. right after your <form> tag that involving POST action, add {% csrf_token %}
3. in the view associated with this POST request, add context_instance=RequestContext(request) as a parameter in your http response.

In the view, when received a POST request, I will generate List objects according to POST parameters. Note that for django models, when you create it, django will automatically generate an unique id as primary key so that you don't have to worry about it. When you create objects for your model in admin site, you also don't have to explicitly indicate their ids. Therefore when you want to manipulate ids for your objects, you have to explicitly mention it in your constructor or whatever.

TBD...

Lab project update 2

First, I didn't convert GET to POST as I said before. Turns out it's pretty difficult to use 3rd party POST request to communicate with a django app: django has banned it for safety issues, particularly CSRF attack. It would take a while if one wants to reenable it. Maybe another time I will sit down and get it over in the future.

Today I find another thing though. So Google Places API supports sorted result, i.e., return a list of places that is sorted based on either 'prominence' or 'distance'. Sorted by distance is just what I need because I need to predict where the user is and of course the geographically nearest place is a good start. However, when you use this feature, it requires you to put in at least one of other three options: keyword, name or types. The first two of course do not fit since I don't know where the user is; the last one makes sense only if we include all types it supports (it has a list). Well, unless there is another to sort by distance, I decide to include all types. This takes me 5 minutes using regular expression. But I do not update the function in the android app, therefore places it stores are still sorted by prominence (by default).

Previously the db in my web app only has fields for time, latitude and longitude, since I am gonna find the nearest place for each pair of coordinates, I decide to modify my db to add three fields: place name, place latitude and place longitude. In this case, it is the best time to learn South. South is a db migration tool for django. Db migration is to let you modify db attributes without wiping all current data. Use db is very simple and it has a great doc support. But one thing though, heroku has three environments, every one needs its own migration, but alway you sync between them. Therefore, you have to be careful that all dbs should be in the same stage as you develop. Otherwise, things could get pretty ugly.

Last thing is I finish the web app with functions to draw the original point, the nearest place obtained from google places, remove points and their places and some nice UI from twitter bootstrap. Twitter bootstrap is such an awesome project that no one would realize its awesomeness until you render your site and play with it. Currently my web app is just look-able, I will dig more from this bootstrap later.

Lab project update 1

Today I work on the project for my summer intern in the lab. Previously, I build it so that it could record and store user's location in the local db periodically in the background, also make it work with google places API and 4sq API. Since we are aiming at "guess" user's semantic location (like in a shop, in a campus, etc) based on their coordinates (either from GPS or cellular or wifi), although we haven't come up with a cool algorithm to guess, it would be better to draw our initial guess on the map so we could have a feel about it.

Instead of integrating a google map with the current android app, I choose to build a web app and let the android app send data to the web app so that it would draw the map. The major reason is we won't need a map, live or not, in the app; it would be overkill. The map is more for analyzing and improving our guess algorithm, not a part of implementation. Besides, I am more confident on my python skill in writing algorithms than Java.

Therefore, my second heroku app born with duty. I still use django. This app is expected to be able to receive data from http request ( generated from the android app), store into its own db, then draw markers using such data on google map. That's it. Might improve later but first let's get it going.

For the http communication part, I look into receiving http request first. It's easy in web framework because they are made to receive http request. For data transmission, coordinates, specifically, I choose to add them into the request url on the sender side. Then when the web app receives the url it will parse it and find correct data:
This method is used as the function described above. Track is the model I create to store coordinates data. After parsing, it will redirect to the home page, if necessary, because the request is only for transmitting data, not displaying web page. Yes you could yell at me because I use GET instead of POST to transmit data. But again, first I want to get it going. GET is fast to implement that is why. I promise I will fix it and return you a long long post about how good and safe to use POST instead of GET.

After this on the android app side, there are a lot of tutorials to tell you how to setup a http request in android/Java using bundled apache libs. Anyway, I insert the request function in my service, so that every time it updates the coordinates it will also send them to my web app.

So that's the first part. How to draw them on the map? after some googling I realize the fact that there is simply no easy and once-for-all effort to create a google map in python. Then I just use js API in my django app. Luckily google has a detailed dev guide. Another good news is, as long as you keep your js script within the html file, you could use django template language to call variables of your app in the js script; it will not work in an independent js file. The rest of story is easy, just do a for loop to extract each pair of coordinate out from the db and create new marker.

I am gonna test it maybe this weekend, after I make some improvements.

heroku migration

UPDATE: I setup a template project for django project bootstrapping on heroku; take a look and fork it.

Finally got my original site up. Phew...

Today I continue my migration work. The major problem is, again, serving static files. I am not sure if other web frameworks have such problem in serving static files too, but this one really bugs me, every time.

Anyway, searched a lot, not much helpful, stuck with it for quite some time. During this I find a weird question though, that people use different project structure for django. As default, when you run startproject command, you will create a project like this:

project_name:
|-- manage.py
|-- project_name:
    |-- __init__.py
    |-- settings.py
    |-- wsgi.py
    |-- urls.py

It creates a subdir inside your project dir with the same name to store configuration files. Somehow people tend to move them up to the project dir and delete the subdir. I am not sure why they are doing this but in my opinion it would be better to stay as the original because thus you could separate project configuration files with other miscellaneous ones like Procfile. Actually some posts I find about how to serve static files use such "flat" structure. This actually leads to a difference in settings.py. So in the file we have to indicate the path for static files, templates, media so that django knows where to find those files. You could of course hard code them if you know every bit of the path for your files, but an easier way would be using a built-in python module os:

import os

SITE_ROOT = os.path.dirname(os.path.realpath(__file__))

STATICFILES_DIRS = (
    ...
    os.path.join(SITE_ROOT, '../static'),
)

TEMPLATE_DIRS = (
    ....
    os.path.join(SITE_ROOT, '../templates')
)

The method above would return the absolute path for this particular file, then we could append specific folder name for different use. This is especially useful when you are doing web dev, because it is likely that you will have at least two environment, development and production. To hardcode for two different environment would be tedious. Now just let the script do the job.

Anyway, you notice the .. before each folder name, because I use the default project structure, my static and templates folders are outside of the folder that contains settings.py. In order to locate them correctly, I have to first go up one level in the dir, then find those folders.

The project structure is not a trivial thing because it will get messy if you have a big project with multiple apps and possibly many external apps. I also find some great tutorials about complex project structure; might need them some day.

Anyway, I keep this original structure, and find a great post about using this structure and Amazon S3 to serve static files for django project in Heroku. It is step-by-step tutorial and most important it works (at least for me). Using S3 is surprisingly simple; now I know why django users all like it. When you setup the environment, every time you add new static files, you use django command collectstatic locally, which will automatically upload new ones to your S3 bucket and serve for your heroku app. Also it will collect admin static files for you as long as you keep them in a child folder admin in your static folder.

After solving static file issue, things get faster. I copy some codes, move some files. Then, I got to say, after shutting down my site for several days, it is good to see it's back live. I mean, I know there is few people would accidentally stumble across it, but for me it is a place I build from the scratch. Now I have this free power of heroku; let's build more.

Hello heroku

Heroku is a web app hosting platform introduced by the SaaS course I take. It's a lot like GAE, but somehow different. For now all I know is it works like a git remote, every time you update/deploy your web app you just push it to heroku remote. One of its biggest feature is scalability: you could start with free account, 5mb/app for storage, no dyno (a work unit in its world), and then choose whatever combination you like to scale up during the development.

I absolutely start with free account. Heroku has a very thorough setup post about django app. But before django I come across this blog about python development environment. It suggests to use virtual environment for every python project I do, because different project might require different version of one package. Make sense. Therefore I use this virtualenv package, followed the excellent-written blog to build up a virtual development for my django project. Virualenv is very easy to use, it comes with pip: every time you setup a completely clean environment, then add packages you want for this particular project using pip.

Done configuring the virtual environment, I start doing this heroku app. Detailed steps are in the tutorial post of heroku. During the setup I meet this problem with ssh keys. It killed me. I have an ssh key previously for github, I didn't realize that its owner has been changed into root. Then when I tried to do git push using this key I keep getting error saying Permission denied (without sudo, for ssh normally you don't need sudo). It took me a while to figure out the root problem from the fact that I could not regenerate the key. Anyway, finally I chown the key also ~/.ssh folder back to my control and regenerate the key.

After that I got a strange problem saying:

-----> Heroku receiving push
 !     Heroku push rejected, no Cedar-supported app detected

I find I misplaced my git into a subdir of my django project. Also, the requirements.txt generated by the virtualenv should be in the git dir. Now everything is done. My first Heroku app.

In the heroku tutorial I find this link to a collection of .gitignore files on github, which is used to ignore useless/auto-generate files of your project during git commit. It provides ignore files fore different languages, very handy.

Heroku use this push strategy, making me feel like coding at home, because I own the environment. Now I could install whatever python packages I like and then push them all up. Beside it has a series of very detailed tutorials on how to get started with different app frameworks also how to use heroku toolset. I think hosting on heroku is not a bad idea. Tomorrow I will try to get back my site on it.

See you AMO

Today I cancelled my tiny plan on the web hosting service A small orange. Lets sum up and move on.

To be honest, AMO provides a decent price, complete toolset, and especially, very responsive helping desk. You know the pain to set up a django project in production (and now I know too), I come across several problems during the setup; guys at AMO gave me a lot help. Thank you all.

But back to the setup, compared with webFaction, my first try, AMO is not optimized for django. It does have a step-by-step wiki for us (which is very detailed and clear), but it does not cover the whole set. I mean there is a chance you could succeed following the wiki, but situations are different in different cases. Moreover, it does not address the static file issue. Django itself does not have the ability to let the server know where to find static files it needs, js, css, pics, etc. The final solution I used is to put all static files under the default dir where I used to serve my static pages. It does not seem to be instinctive because it is separated from the django project. Anyway, I am still new to django so we will see.

At last, anyone likes to build their own website I still recommend AMO. Its tiny plan is a great start ($35/yr for 250MB storage); not to mention their super helpful support guys. Maybe when I try all services online, I will go back to them.