heroku migration

UPDATE: I setup a template project for django project bootstrapping on heroku; take a look and fork it.

Finally got my original site up. Phew...

Today I continue my migration work. The major problem is, again, serving static files. I am not sure if other web frameworks have such problem in serving static files too, but this one really bugs me, every time.

Anyway, searched a lot, not much helpful, stuck with it for quite some time. During this I find a weird question though, that people use different project structure for django. As default, when you run startproject command, you will create a project like this:

project_name:
|-- manage.py
|-- project_name:
    |-- __init__.py
    |-- settings.py
    |-- wsgi.py
    |-- urls.py

It creates a subdir inside your project dir with the same name to store configuration files. Somehow people tend to move them up to the project dir and delete the subdir. I am not sure why they are doing this but in my opinion it would be better to stay as the original because thus you could separate project configuration files with other miscellaneous ones like Procfile. Actually some posts I find about how to serve static files use such "flat" structure. This actually leads to a difference in settings.py. So in the file we have to indicate the path for static files, templates, media so that django knows where to find those files. You could of course hard code them if you know every bit of the path for your files, but an easier way would be using a built-in python module os:

import os

SITE_ROOT = os.path.dirname(os.path.realpath(__file__))

STATICFILES_DIRS = (
    ...
    os.path.join(SITE_ROOT, '../static'),
)

TEMPLATE_DIRS = (
    ....
    os.path.join(SITE_ROOT, '../templates')
)

The method above would return the absolute path for this particular file, then we could append specific folder name for different use. This is especially useful when you are doing web dev, because it is likely that you will have at least two environment, development and production. To hardcode for two different environment would be tedious. Now just let the script do the job.

Anyway, you notice the .. before each folder name, because I use the default project structure, my static and templates folders are outside of the folder that contains settings.py. In order to locate them correctly, I have to first go up one level in the dir, then find those folders.

The project structure is not a trivial thing because it will get messy if you have a big project with multiple apps and possibly many external apps. I also find some great tutorials about complex project structure; might need them some day.

Anyway, I keep this original structure, and find a great post about using this structure and Amazon S3 to serve static files for django project in Heroku. It is step-by-step tutorial and most important it works (at least for me). Using S3 is surprisingly simple; now I know why django users all like it. When you setup the environment, every time you add new static files, you use django command collectstatic locally, which will automatically upload new ones to your S3 bucket and serve for your heroku app. Also it will collect admin static files for you as long as you keep them in a child folder admin in your static folder.

After solving static file issue, things get faster. I copy some codes, move some files. Then, I got to say, after shutting down my site for several days, it is good to see it's back live. I mean, I know there is few people would accidentally stumble across it, but for me it is a place I build from the scratch. Now I have this free power of heroku; let's build more.

Heroku migration pt1

I got an app work on heroku yesterday. But it is just a part of the story. Let's dig a little bit deeper.

Some useful commands. You should always use them to check the status of your app before you want to modify anything. First is to check the basic information of your app in the root dir:

(your_virtualenv_name)your_device:hellodjango your_username$  heroku info
=== afternoon-sword-7524
Addons:        Shared Database 5MB
Database Size: 296k
Git URL:       git@heroku.com:afternoon-sword-7524.git
Owner Email:   your_email
Repo Size:     85M
Slug Size:     12M
Stack:         cedar
Web URL:       http://afternoon-sword-7524.herokuapp.com/

This lists the name, add-ons, database, etc. Give you a general look.

Second is to check all your processes for this app, or as heroku calls, dynos. There are two types of dynos, web dyno, which dealing with web request; work dyno, dealing with your assignment/scheduled jobs for your app. More web dynos means better performance and faster respond when there are a lot of requests to your app; more work dynos should facilitates your work (IMO). When you create a web app on heroku it automatically assign one web dyno for you for free. After that, they will approximately charge you $35/mo for 1 additional dyno. You could see I only assign 1 web dyno for my app:

(your_virtualenv_name)your_device:hellodjango your_username$ heroku ps
=== web: `gunicorn hellodjango.wsgi -b 0.0.0.0:$PORT`
web.1: up for 39m

Third is to check the log of your web app, see what did it do:

(your_virtualenv_name)your_device:hellodjango your_username$  heroku logs
...
2012-06-18T20:23:33+00:00 heroku[web.1]: State changed from starting to up
2012-06-18T20:23:35+00:00 heroku[web.1]: Process exited with status 0
2012-06-18T20:23:42+00:00 heroku[run.1]: State changed from created to starting
2012-06-18T20:23:44+00:00 heroku[run.1]: Starting process with command `python manage.py syncdb`
2012-06-18T20:23:44+00:00 heroku[run.1]: Awaiting client
2012-06-18T20:23:45+00:00 heroku[run.1]: State changed from starting to up
2012-06-18T20:23:47+00:00 heroku[run.1]: Process exited with status 0
2012-06-18T20:23:47+00:00 heroku[run.1]: State changed from up to complete
...

Then we have heroku keys/apps/addons to check our ssh keys, all available apps and add ons for this particular app respectively. in general, it is easy and convenient to check basic information of your apps using heroku CLI tool.

Now for the django project, tutorial gives details so I will not repeat it. Just a problem I come across: work with local database using postgreSQL.

As mentioned in the tutorial, we should use postgreSQL as the production database and actually when we create the app heroku automatically create that db for us. The problem is how to make the app know this is the db it should use. Tutorial use the dj_database_url package which sadly not work for me. I find this postgresify very easy to use. It only needs two steps to set the production db:

from postgresify import postgresify

DATABASES = postgresify()

Now we have the production db but how about local development db? The postgresify seems will not detect default one. When I use it alone I get various error. According to this post, turns out I have to override the production db into local db if django finds it is in the development environment. I use the code form the post and it works finally.

Until now I still cannot restore my site on heroku; it might take longer time.

Hello heroku

Heroku is a web app hosting platform introduced by the SaaS course I take. It's a lot like GAE, but somehow different. For now all I know is it works like a git remote, every time you update/deploy your web app you just push it to heroku remote. One of its biggest feature is scalability: you could start with free account, 5mb/app for storage, no dyno (a work unit in its world), and then choose whatever combination you like to scale up during the development.

I absolutely start with free account. Heroku has a very thorough setup post about django app. But before django I come across this blog about python development environment. It suggests to use virtual environment for every python project I do, because different project might require different version of one package. Make sense. Therefore I use this virtualenv package, followed the excellent-written blog to build up a virtual development for my django project. Virualenv is very easy to use, it comes with pip: every time you setup a completely clean environment, then add packages you want for this particular project using pip.

Done configuring the virtual environment, I start doing this heroku app. Detailed steps are in the tutorial post of heroku. During the setup I meet this problem with ssh keys. It killed me. I have an ssh key previously for github, I didn't realize that its owner has been changed into root. Then when I tried to do git push using this key I keep getting error saying Permission denied (without sudo, for ssh normally you don't need sudo). It took me a while to figure out the root problem from the fact that I could not regenerate the key. Anyway, finally I chown the key also ~/.ssh folder back to my control and regenerate the key.

After that I got a strange problem saying:

-----> Heroku receiving push
 !     Heroku push rejected, no Cedar-supported app detected

I find I misplaced my git into a subdir of my django project. Also, the requirements.txt generated by the virtualenv should be in the git dir. Now everything is done. My first Heroku app.

In the heroku tutorial I find this link to a collection of .gitignore files on github, which is used to ignore useless/auto-generate files of your project during git commit. It provides ignore files fore different languages, very handy.

Heroku use this push strategy, making me feel like coding at home, because I own the environment. Now I could install whatever python packages I like and then push them all up. Beside it has a series of very detailed tutorials on how to get started with different app frameworks also how to use heroku toolset. I think hosting on heroku is not a bad idea. Tomorrow I will try to get back my site on it.