Page:  
Jun
10

Useful Tip for Port Forwarding Apache (or other Web Server)

Problem

You are working on a remote web server where you do not have have access to view the web server’s content on port 80. In my case I only had SSH shell access over port 22.

Solution

Use SSH’s port forwarding capability to serve the content locally on localhost.

    sudo ssh -L 80:name_of_ip_of_webserver:80 username@name_or_ip_of_server_with_access_to_webserver
   

Then just provide the local root password and the password for username@name_or_ip_of_server_with_access_to_webserver. Make sure you stop Apache or other web server before doing this otherwise, this will fail because the port will not be able to bind to a port with an existing bound service. So you might need to do something like this.

    sudo apache2ctl stop
   

Now all you need to do is point your favorite web browser to: http://127.0.0.1/

May
8

How to get SASS going on Ubuntu 10.04

I found that “Ruby 1.8″, packaged with Ubuntu 10.04, does not seem to run SASS out of the box. This is my recipe, cobbled together from various websites, to get things going. I’m using Ruby Version Manager “rvm” to use an updated version of Ruby.

    sudo apt-get install curl git-core ruby
    bash -s stable < <(curl -s https://raw.github.com/wayneeseguin/rvm/master/binscripts/rvm-installer )
   

In ~/.bashrc change you need to replace the line that says:

   [ -z "$PS1" ] && return
   

with

   if [[ -n "$PS1" ]]; then
   

Then add this to the bottom:

    if [[ -s $HOME/.rvm/scripts/rvm ]] ; then source $HOME/.rvm/scripts/rvm ; fi
    fi   
   

Save. Then re-read the .bashrc file.

    source ~/.bashrc  
   

Then from the command line:

   rvm notes
   sudo apt-get install build-essential bison openssl libreadline5 libreadline-dev curl git-core zlib1g zlib1g-   dev libssl-dev vim libsqlite3-0 libsqlite3-dev sqlite3 libreadline-dev libxml2-dev git-core subversion autoconf
   rvm list known
   rvm install 1.9.2-head
    
   #Verify ruby is installed
   ruby -v
   

Now that we have ruby installed lets fetch compass.

    rvm 1.9.2-head
    gem install compass

Phew! Okay now we should be able to watch the SASS directory for changes.
Navigate to wherever your SASS lives and

    compass watch ./sass

Now compass and SASS should be working :-)

May
3

How to Delete a Remote Branch on Github

Sometimes git is hard. An example is deleting a remote branch. Its hard to understand and remember so that’s why this blogpost.

    git push origin :name_of_branch_to_delete
   

It literary means push nothing into branch “name_of_branch_to_delete” on origin (Github)”. Make sure to pay attention to the position of the colon.

Apr
30

Exploring DocNPI.com’s API

DocNPI is a website and webervice for searching information in the National Provider Index (NPI) Registry. The NPI Registry is maintained by HHS and is a registry of all physicians in the United States. The intersting part of DOCNPI for me is that it lets you fetch the information as JSON or CSV thus making it possible to query for this information and import it into other data sets as needed.
(more…)

Oct
14

Fetching Django’s Model’s ‘verbose_name’ and/or Form’s ‘label’ Attribute

Problem:

You want to display some information in a database record in Django, but you also want to display the field’s verbbose name.  In other words, you want to display “What is your favorite Color?” instead of the field name, “favorite_color” or the slightly better “Favorite Color”.  This can be a particular problem if you have very cryptic field names for some reason or another.

The problem is that in Django the “pretty label” information can be defined in the Model’s fields as “verbose_name” or possibly in a related Form as “label”.  In my case I was using ModelForms to make life easier. ModelForms, among other things, take the “verbose_name” values from your Models and stuff it into the Form’s fields “label” values.  You could just create a ModelForm based on your Model, and then fetch the label, but the Label is not defined in the Form if a verbose_name was not defined for the model.  This grabs the pretty label if it exists, and falls back to the field name if its undefined.  This problem is also discussed on Stack Overflow as:.

Django – Iterate over model instance field names and values in template

I’ve provided this solution there as well.

Solution:

There should really be a built-in way to do this. I wrote this utility build_pretty_data_view that takes a model object and form instance (a form based on your model) and returns a SortedDict.

Benefits to this solution include:

  • It preserves order using Django’s built-in SortedDict.
  • When tries to get the label/verbose_name, but falls back to the field name if one is not defined.
  • It will also optionally take an exclude() list of field names to exclude certain fields.
  • If your form class includes a Meta: exclude(), but you still want to return the values, then add those fields to the optional append() list.

To use this solution, first add this file/function somewhere, then import it into your views.py.

utils.py


#!/usr/bin/env python
# -*- coding: utf-8 -*-
# vim: ai ts=4 sts=4 et sw=4
from django.utils.datastructures import SortedDict

def build_pretty_data_view(form_instance, model_object, exclude=(), append=()):
    i=0
    sd=SortedDict()

    for j in append:
        try:
            sdvalue={'label':j.capitalize(),
                     'fieldvalue':model_object.__getattribute__(j)}
            sd.insert(i, j, sdvalue)
            i+=1
        except(AttributeError):
            pass

    for k,v in form_instance.fields.items():
        sdvalue={'label':"", 'fieldvalue':""}
        if not exclude.__contains__(k):
            if v.label is not None:
                sdvalue = {'label':v.label,
                           'fieldvalue': model_object.__getattribute__(k)}
            else:
                sdvalue = {'label':k,
                           'fieldvalue': model_object.__getattribute__(k)}
            sd.insert(i, k, sdvalue)
            i+=1
    return sd

So now in your views.py you might do something like this

from django.shortcuts import render_to_response
from django.template import RequestContext
from utils import build_pretty_data_view
from models import Blog
from forms import BlogForm
.
.
def my_view(request):
   b=Blog.objects.get(pk=1)
   bf=BlogForm(instance=b)
   data=build_pretty_data_view(form_instance=bf, model_object=b,
                        exclude=('number_of_comments', 'number_of_likes'),
                        append=('user',))

   return render_to_response('my-template.html',
                          RequestContext(request,
                                         {'data':data,}))

Now in your my-template.html template you can iterate over the data like so…

{% for field,value in data.items %}

    <p>{{ field }} : {{value.label}}: {{value.fieldvalue}}</p>

{% endfor %}

Good Luck. Hope this helps someone!

Solution

Aug
19

OMHE Microsyntax for Medical Device Interoperability: An Open-Source Alternative to the Continua Alliance

OMHE (Open Mobile Health Exchange), pronounced “ooommm” is an open-source microsyntax for medical devices, mobile text messaging, Twitter®, smart phones, and any system capable of sending a short text string. OMHE’s goal is to provide device interoperability between systems that transmit or receive fitness and wellness information. Its designed for use by both humans and machines. Unlike XML or JSON its simple enough that in most cases a person can type or read OMHE. OMHE still has enough syntactical structure to be parsed and used by machines or software. Another key design decision in OMHE is that OMHE does not define the mode of transport, but simply the data itself. This means OMHE can be blurted out by an electronic weight scale or sent from a human via mobile text message.

OMHE was developed by group of health hackers interested in making it easier to create systems that communicate health information. As an open source specification, there is no license or club to join. Unlike the Continua Alliance which costs thousands to join and who’s utility is questionable, OMHE is a grassroots project defining a practical solution to interoperability. Those interested in contributing code, content, time or funding to the OMHE project should first request to join the Google Group at http://groups.google.com/group/omhe_microsyntax. The OMHE project is hosted on Google Code at http://code.google.com/p/omhe/.

Jul
20

Django Production Deployment: How to Make Apache2 mod_wsgi and virtualenv for Python Play Nice

Problem:

You are a good little Python developer and you use virtualenv in your Python development.  If you aren’t using virtualenv you’re doing it wrong. You have one of the follwoing problems/desires.

  • You want/need to use a Python virtual environment in your production deployment to encapsulate your configuration
  • You have multiple Apache virtual hosts on the same machine and/or you are running serveral Python/Django applications with different or conflicting versions of Python packages.

Solution:

I will demonstrate the solution by installing a Pinax application.  Pinax is a website framework buit on top of Django.  Pinax has many dependencies (including a specific version of Django). We will use Pinax version 0.7.3 whcih requires Django 1.0. I found the the official Pinax deployment documentation very weak and it did not work for me out of the box.  Have no fear. The following steps should get you going.  You can also use this blog post as a general tutorial on production deployment of Django applications.  This solution was tested on Ubuntu 10.10 64-bit server edition.

Step 1: Install system pre-requisites:

$ sudo apt-get update
$ sudo apt-get install git-core build-essential python-setuptools python2.6-dev

Step 2: Install `pip`, `virtualenv`, and ‘virtualencwrapper':

 $ sudo easy_install pip

Step 3: Install ‘virtualenv’ and ‘virtualenvwrapper’ with pip:

 $ sudo pip install virtualenv virtualenvwrapper

Step 5: Setup virtualenv wrapper to make dealing with Python virtual environments trivial.

This will let us use the command “workon” and say “workon ve” to switch to the “ve” virtual environment.  All wee need to do is add a couple items in our .baschrc file. You can use any text editor to do this but I will give the commands using “nano”.  Nano is a terminal based text editor pre-installed with Ubuntu.  “vi” and “emacs” are also options.

$ cd ~ 
$ nano .bashrc

Add the following lines to the bottom of the file and then save it by typing CTL-x and then hit ‘y’ to confirm the save.

export WORKON_HOME=$HOME/.virtualenvs

source /usr/local/bin/virtualenvwrapper.sh

Now activate virtualenv wrapper by re-running the .bashrc

$ source .bashrc

Step 6: Download and install Pinax

Before we download Pinax, here is a little background. The Pinax installer will use pip to fetch other prerequisites and create the virtual environment.  This is special to Pinax. Under a non-Pinax configuration, you would download your app, create a virgin virtalenv, activate it, and then run pip to fetch prerequisites and install them to the new virtualenv.  For clarity and completeness I’m going to illustrate the these basic steps here. I’ll use an imaginary project name “foobar” (You don’t need to do this for Pinax):

$ wget http:/somewebsite/foobar.tar.gz
$ tar zxvf foobar.tar.gz
$ mkvirtualenv foobar-env --no-site-packages
$ workon foobar-env
(foobar-env)$ cd foobar
(foobar-env)$ pip install -r requirements.txt

This will install the prerequisites in the file “requirements.txt” to the virtual environment named “foobar-env”.

Okay back to the task at hand.  Let’s install Pinax. Download and untar/unzip Pinax:

$ wget http://downloads.pinaxproject.com/Pinax-0.7.3-bundle.tar.gz
$ tar zxvf Pinax-0.7.3-bundle.tar.gz
$ cd Pinax-0.7.3-bundle/

Create the virtualenv and install the prerequisites.  I’m calling my virualenv “pinax-env”.   This step could take a while.

$ python scripts/pinax-boot.py $WORKON_HOME/pinax-env

Activate (i.e. switch to) the virtualenv.

$ workon pinax-env

Next let’s create a directory called projects to hold our actual project

(pinax-env)$ cd ~
(pinax-env)$ mkdir projects
(pinax-env)$ cd projects

Now lets clone a new Pinax project based on the Pinax intranet_project base. The first line will display a list of all project types.

(pinax-env)$ pinax-admin clone_project -l
(pinax-env)$ pinax-admin clone_project intranet_project myintranet

Install Python Imaging Library (PIL) which seems to be missing from Pinax installer

(pinax-env)$ pip install PIL

Step 7 : Fire Up the Pinax Project

(pinax-env)$ cd myintranet/

(pinax-env)$ python manage.py syncdb

(pinax-env)$ python manage.py runserver

Point your browser at http://localhost:8000/ and you should see the Pinax default homepage.  If you are hosting remotely you may want to do the following:

(pinax-env)$ python manage.py runserver 0.0.0.0:8000

This will make Django serve over the Internet.  Note this is only development server.   We are just verifying things are working up to this point.  Now here is where it gets insteresting.  This blog post is about deployment on Apache!  Now we want to make install Apache 2, mod_wsgi, and tell Apache to serve the Pinax/Django project located at “~/projects/myintranet” using our virtualenv called “pinax-env.  I’m using “ubuntu” as the deault username.  So our home directory is “/home/ubuntu/”, our project directory is “/home/ubuntu/projects/myintranet” and the root of our virtalenvs is “/home/ubuntu/.virtualenvs/”.  Change this as needed to suit your system.

Step 8: Install Apache2 and mod_wsgi.

Hint don’t install mod_python too.  This can mess things up.

$ sudo apt-get install apache2 libapache2-mod-wsgi

Step 9: Configure Apache to serve our project

We need to tell Apache to server our project.  We do this by pointing Apache to a special  python-based WSGI configuration file for our project. For Pinax, a sample file is located at “deploy/pinax.wsgi” . For Apache, the main file you need to edit is “/etc/apache2/sites-available/default”.  Add the following lines in blue  to “/etc/apache2/sites-available/default”:

<VirtualHost *:80>
ServerAdmin webmaster@localhost
DocumentRoot /var/www
Deny from all
Allow from 127.0.0.0/255.0.0.0 ::1/128
</Directory>
<VirtualHost *:80> ServerAdmin webmaster@localhost DocumentRoot /var/www <Directory /> Options FollowSymLinks AllowOverride None </Directory> <Directory /var/www/> Options Indexes FollowSymLinks MultiViews AllowOverride None Order allow,deny allow from all </Directory> WSGIScriptAlias / /home/ubuntu/projects/myintranet/deploy/pinax.wsgi <Directory /home/ubuntu/projects/myintranet/deploy> Order deny,allow Allow from all </Directory> ScriptAlias /cgi-bin/ /usr/lib/cgi-bin/ <Directory "/usr/lib/cgi-bin"> AllowOverride None Options +ExecCGI -MultiViews +SymLinksIfOwnerMatch Order allow,deny Allow from all </Directory> ErrorLog ${APACHE_LOG_DIR}/error.log # Possible values include: debug, info, notice, warn, error, crit, # alert, emerg. LogLevel warn CustomLog ${APACHE_LOG_DIR}/access.log combined Alias /doc/ "/usr/share/doc/" <Directory "/usr/share/doc/"> Options Indexes MultiViews FollowSymLinks AllowOverride None Order deny,allow Deny from all Allow from 127.0.0.0/255.0.0.0 ::1/128 </Directory>

Now its time to complete our configuration by editing “home/ubuntu/projects/myintranet/deploy/pinax.wsgi”.  Pinax.wsgi file didnt work for me but the configuration I gleamed from mod_wsgi’s wiki on Virtual Environments will work.  Completely erase the default pinax.wsgi file and replace it with the following:

ALLDIRS = ['/home/ubuntu/.virtualenvs/pinax-env/lib/python2.6/site-packages',
'/home/ubuntu/projects',]

import os
import sys
import site

# redirect sys.stdout to sys.stderr for bad libraries like geopy that uses
# print statements for optional import exceptions.
sys.stdout = sys.stderr
prev_sys_path = list(sys.path)

for directory in ALLDIRS:
  site.addsitedir(directory)

# Reorder sys.path so new directories at the front.
new_sys_path = []
for item in list(sys.path):
    if item not in prev_sys_path:
        new_sys_path.append(item)
        sys.path.remove(item)
sys.path[:0] = new_sys_path 

activate_this = '/home/ubuntu/.virtualenvs/pinax-env/bin/activate_this.py'
execfile(activate_this, dict(__file__=activate_this))
from os.path import abspath, dirname, join
from site import addsitedir

sys.path.insert(0, abspath(join(dirname(__file__), "../../")))

from django.conf import settings
os.environ["DJANGO_SETTINGS_MODULE"] = "myintranet.settings"

sys.path.insert(0, join(settings.PINAX_ROOT, "apps"))
sys.path.insert(0, join(settings.PROJECT_ROOT, "apps"))

from django.core.handlers.wsgi import WSGIHandler
application = WSGIHandler()

The key things that may change if the project path and/or virtual environment change are highlighted in blue.  Note you want to remove the following lines if you are setting up a regular Django project (one that is not based on Pinax).

sys.path.insert(0, join(settings.PINAX_ROOT, "apps"))
sys.path.insert(0, join(settings.PROJECT_ROOT, "apps"))

Step 10: Restart Apache and Cross your fingers

$ sudo apache2ctl restart

If the god s are smiling on you today you should be in business.  check by going to http://localhost/.  If not, you might want to check Apache’s error log.

$ cat /var/log/apache2/error.log

There are many other Django and Pinax specific setting you will likely want to change.  Please see the Django Project and the Pinax Project for more information. Good luck. :-)

Jul
1

HTTPS / SSL Configuration Checklist

This blog entry describes how to configure mod_ssl on Ubuntu. This is designed as a quick reference for configuration.

   sudo -i
   openssl genrsa 1024 &gt; www.mysite.com.key
   openssl req -new -key www.mysite.com.key &gt; www.mysite.com.csr</pre>
   

Make sure the “Common Name” is your host name.  In this example its “www.mysite.com”

You can then give your CSR file to rapidssl, register.com, or other certificate provider.

To create a self-signed certificate “”a.k.a. snake oil” you can do the following:

sudo openssl x509 -req -days 365 -in www.mysite.com.csr -signkey www.mysite.com.key -out www.mysite.com.crt

Instead of creating a snake oil certificate, here is where you can go buy a certificate. Replace the SSLCACertificateFile with the one you get from the certificate vendor.

In /etc/apache2/sites-available/default-ssl. Make sure to add the ServerName directive. Make sure your SSL certs point to real files that exist.  See example:


  <IfModule mod_ssl.c>;
  <VirtualHost _default_:443>
  ServerAdmin webmaster@localhost
  ServerName www.mysite.com
  DocumentRoot /var/www
  <Directory>
 ..
 ..
 #   SSL Engine Switch:
 #   Enable/Disable SSL for this virtual host.
 SSLEngine on
 SSLCertificateFile /etc/apache2/ssl/www.mysite.com.crt
 SSLCertificateKeyFile /etc/apache2/ssl/www.mysite.com.key
 SSLCACertificateFile /etc/apache2/ssl/www.mysite.com.crt
 ..
 ..
</VirtualHost>
</IfModule>

Also make sure you enable the ssl and the default-ssl site configuration!

sudo a2enmod ssl
a2ensite default-ssl
/etc/init.d/apache2 reload

Jun
15

How to Encrypt an EBS Volume (Disk) on Amazon EC2

Problem:

You are using Amazon EC2 cloud computer and you want to create an
encrypted EBS volume to store sensitive data.For example, you may have a requirement to
protect ‘data at rest’ if you are dealing with personal health information (PHI) in the United States.

These instructions have been tested on Ubuntu 10.10 64-bit Server Edition.

Solution:

1.) Create an EBS disk and attach it to your instance in the AWS management console.

2.) Attach the device to an open /dev device.  In this example we will use ‘/dev/sdc’.

3.) On your EC2 instance perform the following steps.

    sudo apt-get install -y cryptsetup
    sudo modprobe sha256
    sudo modprobe dm_crypt
    sudo modprobe dm_mod
    sudo modprobe aes
    sudo cryptsetup luksFormat -c aes -h sha256 /dev/sdc
    #type YES and return enter a strong passphrase
    sudo cryptsetup luksOpen /dev/sdc encrypted_disk
    #re-enter your passphrase
    sudo mkfs.ext4 /dev/mapper/encrypted_disk
    sudo mkdir /myencryptdisk
    sudo mount -t ext4 /dev/mapper/encrypted_disk /myencryptdisk

And that’s it!  Remember you will need to re-enter the passphrase when remounting the EBS volume.  Good luck!

-Alan @aviars

Mar
29

How to Create a Geospatial Query with maxDistance in MongoDB using Python

My company (Videntity) has been providing support to the Earth Institute (Columbia University). One of the tasks is to build a geospatial web service for tracking health facilities in the developing world. Its called Georegistry.org. You can think of it like a back-end database for displaying information in GoogleMaps and other mapping software. You can say things like, “Show me all the religious primary schools in Miga (Nigeria)” or “Show me all the water points in Ghana”. It returns data in GeoJSON format which is easy for many different Geographic Information Systems (GIS) to use as input.

Its a bit like simplegeo.com, but its completely open source. Georegistry.org is still a work in progress but coming together nicely. You can download the source code here.  Anyhow I needed to support a geospatial search query with a maximum distance parameter.  After hours of trial and error and a few Internet hints here and there, I finally got it working.  Apparently, the trick is you need an ordered dictionary. This fact is not (as of this writing) in the MongoDB or pymongo documentation. This may seem simple enough but, alas you cannot order a dictionary (in Python) because a Python dictionary is just a mapping!  So we need to use the pymongo.son.SON package.  SON is a subclass of dict and preserves ordering. You can pass a SON object into pymongo’s find() method just like you would a standard Python dictionary.   Anyhow, here is a little example. I hope this helps someone else who stumbles on this problem. I was pulling my hair out for a few hours.

from pymongo import Connection
from pymongo.son import SON

def near_geoquery_with_max_distance(lat, lon, max_distance):
   """ Build the $near dict with your lat/lon values """
   near_dict = {'$near':[float(lat), float( lon)]}

   """ build the maxdistance dict with your max distance value """
   max_dist_dict={'$maxDistance': int(max_distance)}

   """ Create a SON object from our near_dict.
       ORDER IS IMPORTANT. THIS MUST BE FIRST!
   """
   q=SON(near_dict)

   """ Now add the 2nd item (max_dist_dict) to the ordered SON dict """
   q.update(max_dist_dict)

   """ Now put all of the above into a dict with using the key for your
       geospatial data
   """
   gq={'location': q}

   """ Connect to the DB, use you local settings """
   mconnection =  Connection("MONGO_HOST", "MONGO_PORT")
   db =        mconnection["MONGO_DB_NAME"]
   transactions = db["COLLECTION_NAME"]
   """perform the search.  all should work now."""
   mysearchresult=transactions.find(gq)
   .
   .
   .

Good Luck!

Page: