Multiple AMF Django apps share a common gateway

A while back I wrote about a library called django-flashpolicies for providing a flexible remoting environment. Essentially, it frees up the root of your django project for whatever content you like while providing a crossdomain policy file there for remoting applications.

That is still one way to go, but as an advance on that, have a look at the Django-amf-gateway library.

The PyAMF team offers info on it as a way to structure a django project such that you can heap on all the apps you want and allow them to share a common http gateway. By simplifying gateway addresses on the server side django-amf-gateway simplifies resource management on the client side.

Works great in applicable cases.

MAMP with mod_wsgi

Once you get a database set up, sqlite or otherwise, you can use it with Django applications. You can serve Django directly from a command line or through Apache. In either case you will need Django to be installed. From a command line, simply run the django server and browse to it:

django-admin.py runserver

If your production server is going to be Apache, your best bet is to run Django under Apache locally. To do this you need a helper library for Apache to use. The de facto solution is mod_wsgi.

mod_wsgi

To get started with mod_wsgi, typically you download mod_wsgi from Graham Dumpleton's Google Code repository, you go to your command line, you untar it, you configure it, you make it and you make install it. This makes the code library available to wsgi-enabled services, in effect allowing you to run Django on Apache.

When you make install mod_wsgi, it places the libraries in your /lib directory. Then it links to your Apache directory where OSX keeps it, /usr/libexec/apache2. Installation includes automatic detection, or you can use some optional flags to specify resource paths, detailed in the wsgi installation guidelines.

Now your OSX Apache has modwsgi installed, but the Apache version in MAMP does not. To verify this, try making a Django app and serving it via MAMP. First, make a new django app in an appropriate directory:

django-admin.py startproject myproject

Next, open and modify the config for your MAMP Apache app, located in /Applications/Mamp/conf/apache/httpd.conf. For that matter, have a look at what your OSX Apache instance has in its config file. It is private/etc/apache2/httpd.conf. Searching for wsgi in this file yields:

LoadModule wsgi_module libexec/apache2/mod_wsgi.so

Looks good even if the relative directory location may be an issue. Insert this info into the MAMP httpd.conf and test it. Actually, there is no need because it will not work. This is due to the structure of MAMP and its proprietary use of Apache. Unlike a full install of Apache, the folders in MAMP have a different structure and are optimized to run modularly. Consequently, you cannot append a flag to your configure command in this manner:

./configure --with-apxs=/usr/sbin/apxs2-worker

because the structure available to Apache is not in MAMP, so detection will fail.

This leaves a couple of options. Either reconstruct some of the MAMP Apache folders so detection will succeed or look for another solution. Considering the evident lack of general success from many Google searches, I am letting this fish off the hook. Feel free to leave a comment if you know a way to make it work.

As a final consideration, mod_python appears to be a lot quicker to set up with MAMP. That could be a candidate if you are in a pinch. Otherwise, if serving Python via Apache is your goal, I recommend using the resources included with OS X and skipping MAMP altogether.

RemoteObject Endpoints in Django

A few quick assumptions about your project:

• It's built with a lightweight proxy server such as nginx or lighttpd
  
• It utilizes the Django web framework
• It remotes data using some form of ORM
  
• It is presented via Flash/Flex

If this is the case, the simplest and most straightforward way to configure your gateway is by setting it at the root URL:

#my /etc/nginx/sites-enabled/mysite.conf
server {
         listen                    50.5.5.50:80;
         server_name      myurl.com www.myurl.com;
         if ($host = 'www.myurl.com' )    {
               rewrite ^/(.*)$ http://myurl.com/$1 permanent;
         }

         location / {
               proxy_pass    http://127.0.0.1:80/;
               include            /etc/nginx/proxy.conf;
         }

         location /media/  {
                root                /home/django/domains/myurl.com/public_html/;
         }
}

Thus all web traffic that doesn't match '/media/' is passed to the urls.py for further processing. This is however at odds with your remoting requirements for Flash/Flex. To wit, you need a static file called crossdomain.xml to be available at your site's web root in order to meet the Flash client's security needs.

I looked for a good answer to this and found the Django-Flashpolicies by James Bennett effective and QUICK for this very purpose.

All you will need to do, essentially, is this:

easy_install django-flashpolicies

...and this...

url(r^'crossdomain.xml$',
  'flashpolicies.views.simple',
  {'domains': ['media.example.com', 'api.example.com']}),

...and you're good to go. Works better than a savage karate chop to the nose, most of the time anyway.

Hope that allows you to move forward quickly.

Apache DocumentRoot in Python and Django

Be very deliberate in your overall architecture when setting up new django projects. The directory dependencies resulting from an apache-nginx-python-django server configuration can make your path dependencies very complicated.

Generally put, being indiscriminate in your setup now will result in extreme challenges to resource deployment later on.

Here are some working methods that can greatly complicate your setup:

• Hosting several modules on one URL
• Having only one virtual host for many projects
  
• Hosting several modules on one server account
• Setting Python gateway modules off the root of your URL
• Using Python server.py-type gateways in addition to ordinary http gateways

Here are some ways to simplify your paths:

• Set up several virtual hosts
  
• Host only one Django project per URL (or as few as you can manage)
  
• Clearly structure your resources to discourage any overlap on your VPS or dedicated server
• Consider the trade-offs between a complex rig of sites and resources and several simple unrelated server environments. There's really no need to attempt the most complex possible solution unless that's your bag.

The nice thing about Python and Django is that they can handle either direction. There's a way to work everything out however you prefer, and the choice is yours to make.

That said, remember also that it is nice when projects just work without a lot of complex troubleshooting or traffic control.

Here is an example of a moderately complex configuration with a mixed bag of pros and cons.

Setting up PyAMF for Django on Slicehost VPS

I'm continuing to elaborate on a server environment for multiple django projects. In this case, they are serving flex applications on the front end and providing AMF data through PyAMF.

The structure allows domains to be added as needed in a /domains/ subdirectory. Each directory has a virtualenv set to encapsulate the packages installed and minimize complications.

Currently, I am installing PyAMF onto my current project dir using easy_install, from /domains/site.com/site.com/bin

./easy_install pyamf

it is important to run the command from within your virtual environment, otherwise it will default to your system install at /usr/lib/python2.5/site-packages, which isn't mapped to a server and won't be using pyamf at all.

Install PyAMF 0.42 successful. "So far so what," you say pithily. Well, it could certainly be worse than that. Remember that easy_install allows indiscriminate updating of installs via the command

./easy_install -U pyamf

Next some info about eclipse and subversive.

Eclipse Galileo with PyDev and Subclipse

Just installing a bunch of stuff so I can begin developing some django projects locally for a remote deployment. Consisting of:

• Eclipse 3.5 Galileo
• Subclipse 1.6.x
• Python 2.6
• ActiveState ActivePython (currently 2.6)
  

Getting it up and running.

Update: A WinXP install of Eclipse 3.5 will not be able to svn over ssh (see below instead) until you point out a valid ssh app in its config. A good method for this is on the breakitdownblog.

Get the latest PyDev for Eclipse Galileo at http://pydev.sourceforge.net/updates.

Don't forget, you'll have to manually install the connectors. Galileo will need SVNKit and/or JavaHL to reach your subversion server.

Enabling Bombproof Django Logging For Beginning a Project

"Hey, I installed Django and it doesn't work."
"Don't panic. Check the error logs."
"Logs? Django doesn't do logs."
"Python does. You have to add them into your project."
"Yes, but Django doesn't work. Now I am panicking."

And...cut. Beautifully acted all around. The point of the above is, there is a potential chicken-and-egg scenario brewing when you:

1. Have a nonfunctioning Django app.
2. Install logging to debug it.
3. Can't run the application to test the logging...
4. to run the application.

I poked around a bit in search of cut-n-paste logging scripts that get a project back into the light of day. I found a few, so thanks to lawfulsamurai and caktus for hearty illumination.

For brevity and to add a couple of caveats, I have combined what was published elsewhere into this quick how-to. Simply put, it will get you up and logging straight away. Just add it to your settings.py:

import logging
import logging.handlers

logger = logging.getLogger('project_logger')
logger.setLevel(logging.INFO)

# IMPORTANT, make sure python can write to the below log-file, or you will get an error
LOG_FILENAME = '/root/to/your/logging/dir/and/log-file'

LOG_MSG_FORMAT = '%(asctime)s %(levelname)s %(message)s'

handler = logging.FileHandler(LOG_FILENAME)

formatter = logging.Formatter(LOG_MSG_FORMAT)

handler.setFormatter(formatter)
logger.addHandler(handler)
logger.info('testing logging')

There are some fine debugging options out there e.g. djangologging and the django debug toolbar. The advantage of the above is, it is simply a python module so you can't get more native than that.

Feel free to try whichever you find is best and to comment on this option.

Basic Google App Engine command line

Some basic Python SDK commands in Google App Engine

• dev_appserver.py - the development web server
  
• appcfg.py - for uploading your app to App Engine

To run a local instance of a site named mysite within the Google App Engine localserver, invoke command line:

dev_appserver.py mysite (as opposed to python manage.py runserver)

When it comes to starting a project you can opt for either the built-in 0.96 of Django that is in the SDK or alternately bootstrap a 1.x version of Django. If the built-in is satisfactory, the first step in creating the new app engine site is identical to starting a self-standing Django project:

django-admin.py startproject mysite

If you choose the latter method, you will have more flexibility with the latest features of Django. This requires a different startup. You can simply checkout the latest helper app from google code. This serves as a platform for the applications you plan to create. Here's the command line:

svn checkout http://google-app-engine-django.googlecode.com/svn/trunk/ mysite

So now you're all set with a base. Don't forget that with 1.x of Django, you have to put in the Django library locally, probably best using the zipimport method, the details of which are in an earlier post on this blog. Regardless of which Django version you opt for, creating an application is done thus:

python manage.py startapp myapp

Next, begin connecting the apps as you would in Django standalone, via urls.py. There are also special requirements using the App Engine datastore instead of an relational database. More info on the differences can be found here.

PIL for Google App Engine's Python SDK

If you plan to use images in your Django-framwork Google App, you need to add the Python Image Library. If you are using os x 10.5, this bullet list can help set it up. This assumes you are working with the native Python 2.5 installation rather than a new install. Why install it again? You don't need to.

• Google's PIL documentation is fine but fragmentary.
• The update provided by Matt Kangas is really all you need.
• If you have not modified Terminal to run shell scripts, macfidelity provides details to get the job done.

That should be all that is required.

Google App Engine with Django 1.0

Google App Engine has offered Django 0.96 out of the box since it launched in mid 2008. That has changed as of this month.

Applications can now choose either Django 0.96 or what is currently the latest stable release, 1.0.2. This means there is no need to post a zipfile to your appspot in order to utilize the latest Django. But is it that simple? Not quite.

While the cloud has the latest version, your local server environ does not. In other words, the SDK and appspot server are a bit different. This means you still need the Django Helper to run 1.0.2 locally, where you will probably be developing your site.

I went through this process and found many resources to shed light on the procedure. Some are more suited than others, due to good but out-of-date info on the woodpile. Here's what did work:

• Google Code documentation on the Python SDK - what I like to think of as the GAE personal web server, though it's actually that and a bit more.
• Google documentation on the Django Helper - required reading but terse. Provides and describes a turnkey local project running Django 1.0.2.
• Finally, from Bubble Foundry in Nederland, the paydirt, aka how to get it done, aka as direct as a battleaxe to the head. Thanks, I needed that!

Good luck setting up your dev environment. Google labs makes it not so bad after all.

Django AMF Service

Now that the basic Django framework is in place (admin, views, urls), the focus shifts to data and services. In the past, WebORB has been adept at providing RPC between ASP and RIA, so it's tempting to use it again in a Linux environment.

Since the current project is Apache, one option is to utilize Apache Tomcat. Tomcat is readily available in MediaTemple's (dv) configuration to provide a pure Java HTTP server environment. This would pair well with WebORB for Java, a robust middleware solution for RIA data synchronization.

Good enough but the tiers don't fully align with the planets yet. My goal is to utilize remote procedure calls to mySQL and take advantage of the schema I've built in Python-based Django. That will not be possible until my Python objects can be called from Java. So, is it possible to port Python code to WebORB via a Python -> Java compiler?

In a word, Jython.

Jython runs in Tomcat as an implementation of the Python programming language. By compiling the Django code, Jython could route between Flex on the front end and the mySQL admin system within the framework of Django.

Incidentally, I'm not going to implement this cookbook recipe just yet. I've worked with WebORB and am sold on it's strengths, but in this instance I'm looking for a lighter and more agnostic solution for the frontend than a uniform rich presentation server.

Namely, I'm going to move forward with Django AMF PyAMF. This allows some of Django's front-end capabilities to shine, and also enables javascript and ajax elements to be plugged into the site design.

More posts to come on this direction.

Out of curiosity, has anyone else tried the setup I outlined above, i.e. Django <-> Java <-> RPC tier <-> RIA? Furthermore, would you go a completely different route or change any of the tiers from the architecture I outlined? It is one of countless possibilities, and on that note, in my experience, one of the most direct and stable available.

Django MVC Basics Pt 1: URLconf

The URLconf - mapping between URLs and view functions that should be called for those URLs.

from django.conf.urls.defaults import *

# Uncomment the next two lines to enable the admin:
# from django.contrib import admin
# admin.autodiscover()

urlpatterns = patterns('',
# Example:
# (r'^mysite/', include('mysite.foo.urls')),

# Uncomment the admin/doc line below and add 'django.contrib.admindocs'
# to INSTALLED_APPS to enable admin documentation:
# (r'^admin/doc/', include('django.contrib.admindocs.urls')),

# Uncomment the next line to enable the admin:
# (r'^admin/', include(admin.site.urls)),
)

Important to note are:
r - Python indication that the string is a "raw string", i.e. do not interpret characters.
' ' - string enclosed in single quotes
^ - caret, a regular expression meaning "require that the pattern matches the start of the string"
$ - dollar sign, regex meaning "require the pattern match the end of the string".

e.g. ensures this works: www.url.com/hello/
and this does not: www.url.com/hello/there (thanks to $)
and this does not:www.url.com/there/hello (thanks to ^)

See table of basic Regular Expressions - from the Django Book.

The second parameter in the 2-tuple sometimes begins with include(... and sometimes does not. It is the function to be associated with the specified URL. The function's first parameter is always the HttpRequest object. Always. Here's documentation on the HttpRequest object's API.

Last (?) post about MediaTemple (dv) and Django

This is a refinement of a previous post on the topic of Django integration into a MediaTemple (dv) account. Following a talk with MT and reading some knowledgebase info, here is the quick route to Django configuration happiness - now more better.

This is how we do it:

Add to base config aka /etc/httpd/conf/httpd.conf the following in the appropriate section:

LoadModule wsgi_module modules/mod_wsgi.so

And in the appropriate section, add this too:

NameVirtualHost *:80

#Add a VirtualHost Section for each Name-based virtual server

<VirtualHost>
ServerAdmin email@email.com
ServerName www.URL.com
ServerAlias URL.com *.URL.com

DocumentRoot /var/www/vhosts/URL/httpdocs

<Directory /var/www/vhosts/URL/httpdocs>
Order allow,deny
Allow from all
</Directory>

</VirtualHost>

then save the file and exit the editor.

In the directory of path '/var/www/vhosts/(DOMAIN NAME)/conf' there is a file called 'httpd.include.' It holds unique config info for each virtual domain. It is rebuilt automatically from time to time based on Plesk. Instead of editing that file, make a new one called 'vhost.conf' (if it doesn't already exist) and add the following info to this file:

WSGIDaemonProcess URL.com processes=1 threads=1 display-name=%{GROUP}
WSGIProcessGroup URL.com
WSGIScriptAlias /myapp /home/username/django/scripts/URL.com.wsgi

<Directory /home/username/django/scripts>
Order allow,deny
Allow from all
</Directory>

ErrorLog /var/log/URL.com-error_log
CustomLog /var/log/URL.com-access_log common

Notes on the above, in brief

1. LoadModule is in httpd.conf
2. NameVirtualHost declaration is in httpd.conf
3. virtualhosts make use of /virtualhostname/conf/vhost.conf
4. WSGIScriptAlias declarations are in respective vhost.conf, not in httpd.conf*

* This is because WSGIScriptAlias in httpd.conf didn't work

Once that is configured as you like, simply restart the server in this manner:

/usr/local/psa/admin/sbin/websrvmng --reconfigure-vhost --vhost-name=mt-example.com 

service httpd graceful

Now test your django. With any luck, you will see a screen confirming a correct installation.

Python Image Library (PIL) on MediaTemple (dv)

When you're on MediaTemple (dv) with root access, first get the TAR from here. As of this post, the latest version is 1.1.6.

wget http://effbot.org/downloads/Imaging-1.1.6.tar.gz

Untar in the usual way.

tar zxvf Imaging-1.1.6.tar.gz

Then, go into the created folder and install the package using:

python setup.py install

If all goes well your ssh will echo many installation processes, following which you will have an amazingly easy time working with images via http.

Python and Django in MediaTemple (dv) server

Getting Django to run on a dv server is not as easy as their preconfigured - and cheaper - grid containers. While the shared grid containers are ready to go out of the box, the dv has some server admin changes to get through before you see that welcome

"Hello World".

Specifically, the httpd.conf needs additional lines of code to connect the web root to the Django application. To add a little spin to it, there's also the config file generated by Plesk. That needs some edits as well.

The order of statements in the zz010_psa_httpd.conf file can be a particular obstacle. Seems you have to have the WSGIScriptAlias declarations high up in the page or they aren't acted on properly as the procedure continues. Changing LogLevels to info gives the most detailed view from your logfiles but may not ultimately identify the problems.

In any case, here are the steps in a nutshell:

(added to zz010_psa_httpd.conf - plesk)

Include /var/www/vhosts/urlone.com/conf/httpd.include
Include /var/www/vhosts/urltwo.com/conf/httpd.include
WSGIScriptAlias /myapp /var/www/vhosts/urlone.com/wsgi-scripts/myapp.wsgi
WSGIScriptAlias /django/test01 /var/www/vhosts/urltwo.com/wsgi-scripts/myapp.wsgi

(added both httpd.includes to file structure)

Seems simple enough once it works. I hope this gets you out of a pinch.

Serving Django on Apache

As an update on the previous post, looking into current Django docs, the Apache production server method of using mod_python to serve Django has been superseded by mod_wsgi. I had partially implemented the former method, so now I'm bringing it up to the latest spec.

According to google.code on Python WSGI, Apache 1.3, 2.0 or 2.2 are compatible. Apache on a MediaTemple dv account shows

httpd -v

Server version: Apache/2.2.3
Server built: Nov 12 2008 10:41:27

Python needs to be 2.3 through 2.5 and the vers on (mt) shows

Python 2.4.3 (#1, May 24 2008, 13:47:28)

So far, so what. Now to install wsgi...

wget http://modwsgi.googlecode.com/files/mod_wsgi-2.5.tar.gz
tar xvfz mod_wsgi-2.5.tar.gz
configure
make
make install

finally edit

httpd.conf

located in

/etc/httpd/conf.

Adding the lines:

LoadModule wsgi_module modules/mod_wsgi.so

and


<virtualhost *>
ServerAdmin jasonthewolf@gmail.com
ServerName www.unplugtheinternet.com
DocumentRoot /var/www/vhosts/unplugtheinternet.com
WSGIScriptAlias / /django/test01/apache/django.wsgi
</virtualhost>

In the above, WSGIScriptAlias deserves some explanation:
Description : WSGI maps a URL to a filesystem location and designates the target as a WSGI script.
Syntax : WSGIScriptAlias URL-path file-path|directory-path
Context : server config, virtual host
Module : mod_wsgi.c

A request for http://www.example.com/wsgi-scripts/name in this case would cause the server to run the WSGI application defined in /web/wsgi-scripts/name. This configuration is essentially equivalent to:

and restart apache with

apachectl restart

All reports are good from Apache so far. Now that the installation of wsgi is complete, back to Django implementation.

I once again edited httpd.conf, this time adding
WSGIScriptAlias / /path/to/mysite/apache/django.wsgi

which for my site and requirements was effectively:

WSGIScriptAlias /django/test01 /var/www/vhosts/unplugtheinternet.com/apache/django.wsgi

This is the folder I have chosen for my python tests. If they work out, I'll move them to the root of some url or another. Following the convention established in the example, I created directory apache and the used vi to create the textfile django.wsgi as:

import os
import sys

os.environ['DJANGO_SETTINGS_MODULE'] = 'mysite.settings'

import django.core.handlers.wsgi
application = django.core.handlers.wsgi.WSGIHandler()

Python and Django wrapup

Almost a wrapup.

Got django installed in /usr/lib/python2.4/site-packages/

Got django deployed in /var/www/vhosts/unplugtheinternet.com/django/test01/ with subfolder cms autogenerated

Got MySQLdb module added to python so django can use a MySQL database, and django has been successfully configured to build the schemas for the modules it deploys.

Created a vhost.conf to take care of webserver requests that python should handle (courtesy of Nick Sergeant's blog entry).

Still it isn't working properly. But since I'm leaving for a camping trip in ~1 hour, it's time to switch it off and go join my friends at Assateague Island.

Arrividerci pals.

MediaTemple, Python and Django

Just installing Django on my Mediatemple dv account.

First, log in to SSH as root user. Then use subversion, provided by mediatemple in

WebControl > Root Access & Developer Tools > Install Developer Tools,

which must at some point be enabled to be accessible. Following this,

Subversion grabs the latest version to /var/trunk/django.

In the case of dv accounts, python packages are located in /usr/lib/python2.4/site-packages

Create a symbolic link between these two locations using

ln -s /var/trunk/django /usr/lib/python2.4/site-packages

then add another symbolic link to bring the underlying python script to the root of the django trunk:

ln -s /var/trunk/django/bin/django-admin.py /usr/local/bin

Basically, what is happening here is:

Django Source /var/trunk : Subversion project root

\ --> accessible to /usr/lib : Libraries for programming and packages
\ --> accessible to /usr/local : Local hierarchy

More on this. Now:

import django

and


django.version

voila!

Next, you must add the MySQLdb interface for python if you are planning to use mySQL as your cms database. Here's a good reference for once you have the tarball in your directory. However, I'd rather wget it directly from the sourceforge site. Here is the command to use from root SSH:

wget http://downloads.sourceforge.net/mysql-python/MySQL-python-1.2.2.tar.gz

Works fine. Now install the module and it should run without an error.