Django auth pt1

Nowadays, to use google api correctly in your web applications, most likely you have to use its own authentication oauth 2.0. They say it is simple and easy; to a newbie of web dev it is not. Anyway, I failed to integrate it with my web app, thusly I decide to learn auth from the scratch, and hope one day my web app will be popular enough to demand an authentication system (I mean it!).

Django has its own built-in auth package, which is ideal to use learn the concepts. Most importantly, built-in packages like this always come with built-in views, which could save you a lot of time to get started.

Let's see how to use built-in views: really simple.

In urls.py, we define three extra patterns besides admin ones. One is for the index view, the home page, which will require login authentication to see its content. The other two utilize the built-in views for login and logout. Note for login view you have to define a template to make it go if you don't want to copy the default template from django package; for logout I here use the logout_then_login view, which is pretty self-explained. Since it will redirect to login page automatically, we don't have to add any template for this view.

Now take a look at view.py:

Two things to know: @login_required decorator indicates that this view needs authentication, if currently the user is not logged in, it will take the user to the login view we defined above; if already logged in, it will render the index view as required.

I don't include the template it uses because you could find it in django doc. Only one thing is a little bit tricky: the default login view will pass several variables to the template, including one called next. This contains the url that the app will redirect to after successful login, which should be the url triggered this login (in our case it is "/" for index view). To make the redirect happens, in the template you have to define a redirect to the next, like this if your login page uses a form: <input type="hidden" name="next" value="{{ next }}"/>.

Lastly, remember to add a link to logout url in your index page in order to complete the whole login/logout process.

All done.

GRE helper project update 1

UPDATE: I cut this project. It turns out giving her a list generated by python script is much faster. Not everything needs a web app.

I start yet another django project on heroku today. It's a rather simple one, just to help my girlfriend memorize vocabulary and prepare her GRE test.

GRE is difficult for people whose first language is not English. The test involves so many non-daily English. You have to memorize, like hardcoded in your brain, thousands of words in order to just understand what this test is talking about, not mention other requirements. A typical preparation for GRE involves at least 4 months of memorizing words repeatedly. I have been through this before, it's a very stressful and boring period of time, especially when you spend several weeks and look back, realize you don't remember one damn word. Sadly I could not turn it into a happy thing; however, I think I could make it efficient and maybe shorten its duration somehow.

What I want it to do is let you type in how many vocabulary lists (do you want to know how many words Chinese have to memorize for just a single test?), then it will generate a calendar filled with each list as task, and distributed based on a predefined memory curve. Basically a customizable repetitive task generator hooked up with google calendar. I know it is still dull, but an organized and well-executed repetition on memorizing could efficiently reduce the amount of time you need to achieve your goal. For the calendar, I don't want to reinvent such a good calendar system, besides Google calendar API has a good python support.

Start with the model. It is pretty simple and common, but I intend to add maybe a list as a instance variable so that every list could have a list of dates on which they needs to be memorized. For the progress field, which is supposed to be within 0 to 100, I add two validators from django.core.validators to prevent overflow.


I make just one page, in which you will have a input area for you to type in how many lists you have, and a display area to show the calendar. The input with the submit button will issue a http POST request to server side, along with some parameters, including the number of your lists. Note django itself disable post method automatically to prevent CSRF attack. To reenable it:

1. add 'django.middleware.csrf.CsrfViewMiddleware' to the MIDDLEWARE_CLASSES in your settings.py (this should be done by default in the latest version now)
2. right after your <form> tag that involving POST action, add {% csrf_token %}
3. in the view associated with this POST request, add context_instance=RequestContext(request) as a parameter in your http response.

In the view, when received a POST request, I will generate List objects according to POST parameters. Note that for django models, when you create it, django will automatically generate an unique id as primary key so that you don't have to worry about it. When you create objects for your model in admin site, you also don't have to explicitly indicate their ids. Therefore when you want to manipulate ids for your objects, you have to explicitly mention it in your constructor or whatever.

TBD...

Lab project update 2

First, I didn't convert GET to POST as I said before. Turns out it's pretty difficult to use 3rd party POST request to communicate with a django app: django has banned it for safety issues, particularly CSRF attack. It would take a while if one wants to reenable it. Maybe another time I will sit down and get it over in the future.

Today I find another thing though. So Google Places API supports sorted result, i.e., return a list of places that is sorted based on either 'prominence' or 'distance'. Sorted by distance is just what I need because I need to predict where the user is and of course the geographically nearest place is a good start. However, when you use this feature, it requires you to put in at least one of other three options: keyword, name or types. The first two of course do not fit since I don't know where the user is; the last one makes sense only if we include all types it supports (it has a list). Well, unless there is another to sort by distance, I decide to include all types. This takes me 5 minutes using regular expression. But I do not update the function in the android app, therefore places it stores are still sorted by prominence (by default).

Previously the db in my web app only has fields for time, latitude and longitude, since I am gonna find the nearest place for each pair of coordinates, I decide to modify my db to add three fields: place name, place latitude and place longitude. In this case, it is the best time to learn South. South is a db migration tool for django. Db migration is to let you modify db attributes without wiping all current data. Use db is very simple and it has a great doc support. But one thing though, heroku has three environments, every one needs its own migration, but alway you sync between them. Therefore, you have to be careful that all dbs should be in the same stage as you develop. Otherwise, things could get pretty ugly.

Last thing is I finish the web app with functions to draw the original point, the nearest place obtained from google places, remove points and their places and some nice UI from twitter bootstrap. Twitter bootstrap is such an awesome project that no one would realize its awesomeness until you render your site and play with it. Currently my web app is just look-able, I will dig more from this bootstrap later.

heroku migration

UPDATE: I setup a template project for django project bootstrapping on heroku; take a look and fork it.

Finally got my original site up. Phew...

Today I continue my migration work. The major problem is, again, serving static files. I am not sure if other web frameworks have such problem in serving static files too, but this one really bugs me, every time.

Anyway, searched a lot, not much helpful, stuck with it for quite some time. During this I find a weird question though, that people use different project structure for django. As default, when you run startproject command, you will create a project like this:

project_name:
|-- manage.py
|-- project_name:
    |-- __init__.py
    |-- settings.py
    |-- wsgi.py
    |-- urls.py

It creates a subdir inside your project dir with the same name to store configuration files. Somehow people tend to move them up to the project dir and delete the subdir. I am not sure why they are doing this but in my opinion it would be better to stay as the original because thus you could separate project configuration files with other miscellaneous ones like Procfile. Actually some posts I find about how to serve static files use such "flat" structure. This actually leads to a difference in settings.py. So in the file we have to indicate the path for static files, templates, media so that django knows where to find those files. You could of course hard code them if you know every bit of the path for your files, but an easier way would be using a built-in python module os:

import os

SITE_ROOT = os.path.dirname(os.path.realpath(__file__))

STATICFILES_DIRS = (
    ...
    os.path.join(SITE_ROOT, '../static'),
)

TEMPLATE_DIRS = (
    ....
    os.path.join(SITE_ROOT, '../templates')
)

The method above would return the absolute path for this particular file, then we could append specific folder name for different use. This is especially useful when you are doing web dev, because it is likely that you will have at least two environment, development and production. To hardcode for two different environment would be tedious. Now just let the script do the job.

Anyway, you notice the .. before each folder name, because I use the default project structure, my static and templates folders are outside of the folder that contains settings.py. In order to locate them correctly, I have to first go up one level in the dir, then find those folders.

The project structure is not a trivial thing because it will get messy if you have a big project with multiple apps and possibly many external apps. I also find some great tutorials about complex project structure; might need them some day.

Anyway, I keep this original structure, and find a great post about using this structure and Amazon S3 to serve static files for django project in Heroku. It is step-by-step tutorial and most important it works (at least for me). Using S3 is surprisingly simple; now I know why django users all like it. When you setup the environment, every time you add new static files, you use django command collectstatic locally, which will automatically upload new ones to your S3 bucket and serve for your heroku app. Also it will collect admin static files for you as long as you keep them in a child folder admin in your static folder.

After solving static file issue, things get faster. I copy some codes, move some files. Then, I got to say, after shutting down my site for several days, it is good to see it's back live. I mean, I know there is few people would accidentally stumble across it, but for me it is a place I build from the scratch. Now I have this free power of heroku; let's build more.

Heroku migration pt1

I got an app work on heroku yesterday. But it is just a part of the story. Let's dig a little bit deeper.

Some useful commands. You should always use them to check the status of your app before you want to modify anything. First is to check the basic information of your app in the root dir:

(your_virtualenv_name)your_device:hellodjango your_username$  heroku info
=== afternoon-sword-7524
Addons:        Shared Database 5MB
Database Size: 296k
Git URL:       git@heroku.com:afternoon-sword-7524.git
Owner Email:   your_email
Repo Size:     85M
Slug Size:     12M
Stack:         cedar
Web URL:       http://afternoon-sword-7524.herokuapp.com/

This lists the name, add-ons, database, etc. Give you a general look.

Second is to check all your processes for this app, or as heroku calls, dynos. There are two types of dynos, web dyno, which dealing with web request; work dyno, dealing with your assignment/scheduled jobs for your app. More web dynos means better performance and faster respond when there are a lot of requests to your app; more work dynos should facilitates your work (IMO). When you create a web app on heroku it automatically assign one web dyno for you for free. After that, they will approximately charge you $35/mo for 1 additional dyno. You could see I only assign 1 web dyno for my app:

(your_virtualenv_name)your_device:hellodjango your_username$ heroku ps
=== web: `gunicorn hellodjango.wsgi -b 0.0.0.0:$PORT`
web.1: up for 39m

Third is to check the log of your web app, see what did it do:

(your_virtualenv_name)your_device:hellodjango your_username$  heroku logs
...
2012-06-18T20:23:33+00:00 heroku[web.1]: State changed from starting to up
2012-06-18T20:23:35+00:00 heroku[web.1]: Process exited with status 0
2012-06-18T20:23:42+00:00 heroku[run.1]: State changed from created to starting
2012-06-18T20:23:44+00:00 heroku[run.1]: Starting process with command `python manage.py syncdb`
2012-06-18T20:23:44+00:00 heroku[run.1]: Awaiting client
2012-06-18T20:23:45+00:00 heroku[run.1]: State changed from starting to up
2012-06-18T20:23:47+00:00 heroku[run.1]: Process exited with status 0
2012-06-18T20:23:47+00:00 heroku[run.1]: State changed from up to complete
...

Then we have heroku keys/apps/addons to check our ssh keys, all available apps and add ons for this particular app respectively. in general, it is easy and convenient to check basic information of your apps using heroku CLI tool.

Now for the django project, tutorial gives details so I will not repeat it. Just a problem I come across: work with local database using postgreSQL.

As mentioned in the tutorial, we should use postgreSQL as the production database and actually when we create the app heroku automatically create that db for us. The problem is how to make the app know this is the db it should use. Tutorial use the dj_database_url package which sadly not work for me. I find this postgresify very easy to use. It only needs two steps to set the production db:

from postgresify import postgresify

DATABASES = postgresify()

Now we have the production db but how about local development db? The postgresify seems will not detect default one. When I use it alone I get various error. According to this post, turns out I have to override the production db into local db if django finds it is in the development environment. I use the code form the post and it works finally.

Until now I still cannot restore my site on heroku; it might take longer time.

Hello heroku

Heroku is a web app hosting platform introduced by the SaaS course I take. It's a lot like GAE, but somehow different. For now all I know is it works like a git remote, every time you update/deploy your web app you just push it to heroku remote. One of its biggest feature is scalability: you could start with free account, 5mb/app for storage, no dyno (a work unit in its world), and then choose whatever combination you like to scale up during the development.

I absolutely start with free account. Heroku has a very thorough setup post about django app. But before django I come across this blog about python development environment. It suggests to use virtual environment for every python project I do, because different project might require different version of one package. Make sense. Therefore I use this virtualenv package, followed the excellent-written blog to build up a virtual development for my django project. Virualenv is very easy to use, it comes with pip: every time you setup a completely clean environment, then add packages you want for this particular project using pip.

Done configuring the virtual environment, I start doing this heroku app. Detailed steps are in the tutorial post of heroku. During the setup I meet this problem with ssh keys. It killed me. I have an ssh key previously for github, I didn't realize that its owner has been changed into root. Then when I tried to do git push using this key I keep getting error saying Permission denied (without sudo, for ssh normally you don't need sudo). It took me a while to figure out the root problem from the fact that I could not regenerate the key. Anyway, finally I chown the key also ~/.ssh folder back to my control and regenerate the key.

After that I got a strange problem saying:

-----> Heroku receiving push
 !     Heroku push rejected, no Cedar-supported app detected

I find I misplaced my git into a subdir of my django project. Also, the requirements.txt generated by the virtualenv should be in the git dir. Now everything is done. My first Heroku app.

In the heroku tutorial I find this link to a collection of .gitignore files on github, which is used to ignore useless/auto-generate files of your project during git commit. It provides ignore files fore different languages, very handy.

Heroku use this push strategy, making me feel like coding at home, because I own the environment. Now I could install whatever python packages I like and then push them all up. Beside it has a series of very detailed tutorials on how to get started with different app frameworks also how to use heroku toolset. I think hosting on heroku is not a bad idea. Tomorrow I will try to get back my site on it.

See you AMO

Today I cancelled my tiny plan on the web hosting service A small orange. Lets sum up and move on.

To be honest, AMO provides a decent price, complete toolset, and especially, very responsive helping desk. You know the pain to set up a django project in production (and now I know too), I come across several problems during the setup; guys at AMO gave me a lot help. Thank you all.

But back to the setup, compared with webFaction, my first try, AMO is not optimized for django. It does have a step-by-step wiki for us (which is very detailed and clear), but it does not cover the whole set. I mean there is a chance you could succeed following the wiki, but situations are different in different cases. Moreover, it does not address the static file issue. Django itself does not have the ability to let the server know where to find static files it needs, js, css, pics, etc. The final solution I used is to put all static files under the default dir where I used to serve my static pages. It does not seem to be instinctive because it is separated from the django project. Anyway, I am still new to django so we will see.

At last, anyone likes to build their own website I still recommend AMO. Its tiny plan is a great start ($35/yr for 250MB storage); not to mention their super helpful support guys. Maybe when I try all services online, I will go back to them.