Tutorial: Building a Python-centric WebServer in the Cloud

Version 1.0
by Raymond Hettinger and Mike Ivanov, ActiveState

Introduction

One of the challenges of getting started in cloud computing is starting with an empty instance, learning the Amazon cloud services API, making scalable architectural decisions, and building a complete web stack from scratch.

ActiveState is helping the community by offering a prebuilt AMI with a working web stack and set of tools for customization and deployment.

The toolkit is designed to provide smooth on-ramps at various stages of adoption:

  • Newcomers to cloud computing can have a live website in minutes and use that as a starting point for learning what is involved.
  • Those with existing web pages can drop-in their own HTML and CSS and have them available on-line immediately.
  • The Django web framework offers a slew of pre-built applications that can be used to expand a website’s capabilities with minimal effort.
  • If a website becomes popular, the control script offers simple command-line options for scaling the website across multiple worker instances.
  • Developers needing customizations can use the build and deployment script as a starting point for incremental development, replacing any or all of the components. The stack was designed with loose coupling so that developers can easily use alternative databases like MySQL or PostgreSQL, alternative web-frameworks such as Django or Pylons, alternative templating systems such as Mako or Cheetah, and alternative web servers such as Apache or Gunicorn. The build can also be targeted to various Linux distributions such as Ubuntu or RedHat.

Establishing an Account with Amazon Web Services

The necessary first step for running anything in Amazon’s cloud is to establish an Amazon Web Services account:
https://aws-portal.amazon.com/gp/aws/developer/registration/index.html
To get an overview of how Amazon hosted cloud computing works, see their Getting Started Guide at:
http://docs.amazonwebservices.com/AWSEC2/latest/GettingStartedGuide/

Two Ways to Get Started

After having established an Amazon account and getting access keys, there are two ways to get started with ActiveState’s prebuilt AMI and its build/control scripts.

One choice is to launch your own instance cloned from the working AMI.  This is the fastest way to get a running web server.

You can access, control, and update the instance just like you would with your own desktop computer.  The only difference is that you log-in to the virtual machine using ssh (or Putty in a Windows environment).   With that access, you can copy your own web pages to the /var/www/demosite directory. You can edit the configuration files, check log files, access the database, examine the memcache, or start customizing the web stack.

See the Direct Access section below for a tour of the web stack and the locations of important files.

When doing production work by directly modifying the instance, be sure to periodically save backup copies of the instance.

Another way to use ActiveState’s Amazon AMI toolkit is to develop locally and then create or update instances with the build script that we have supplied on the AMI. The approach provides more flexibility than directly modifying an instance. In particular, using a build script offers important capabilities that are not easily achieved by direct modification:

  • reproducible builds
  • version control over the entire build
  • re-targeting to a smaller API or an alternate operating system
  • command-line control over scaling the website to multiple worker instances
  • adding, removing or updating components from the web stack
  • migrating to alternate hosting on other Amazon availability zones, to a piece of local hardware, to blade servers at a co-location facility, or to another hosting service.

See the Script Driven Development section below for a tour of ActiveState’s customizable build and control script.

The components of the web stack were selected using the principles described in ActiveState’s Best Practices Notes on Cloud Computing with Python. Preference was given to well-known and widely used components that have proven effective. The components are loosely coupled so that alternate tools can be substituted, choosing from various web servers, web frameworks, databases, and caching tools.

The prebuilt AMI starts with a large Amazon instance running a long-term supported 64-bit Ubuntu operating system (version 10.04). The front-end for web traffic is Nginx which serves static content and routes web-traffic to multiple worker instances. The default webserver is Apache (version 2.2.16) running mod_wsgi (version 3.3.7). The programming language interpreter is ActivePython (version 2.6.6 CE upgradable to 2.6.6 BE) and it includes the PyPM packaging manager. The Python components are isolated with virtualenv (1.5.1) and made more user friendly with virtualenvwrapper. The web-server is Django (1.2.3) and it comes with its own Object Relational Mapper. Two databases are provided, SQLite3 (included in the Python distribution) and MySQL (version 5.1). The sample application for the demo website uses Memcached for its high-speed key-value store.

Script Driven Development

The ActiveState cloud development toolkit is run on a local machine. Its only requirements are Python2.6 and two python packages, boto for accessing AWS services and fabric for remote deployment.

You can download the ActiveState cloud development toolkit, from here. In this zip archive, you will find all the scripts, demo application and other files referred to in this tutorial. Unzip the downloaded archive, and start a command line prompt in the unzipped directory to begin this tutorial.

You need to ensure that you have the right version of python installed on your desktop machine. When you start Python you should see “ActivePython” at the very top of the of the startup message. If you don’t have ActivePython installed, download your copy here.

It is a good idea to work in a virtual environment although it is not a requirement of this tutorial.

From here, the simplest way to get up and running is to type these commands to install the only two dependencies for the toolkit:

(env)$ pypm install boto
(env)$ pypm install fabric

The principal parts of the toolkit are in the top level files. The fabfile.py script is a thin wrapper around nimbus/fablib.py which has most of the deployment code -- this is where you will put build customizations. The settings.py is where you specify your configuration options. The toolkit is run from the command-line.

See all of the commands with fab --list.

A short sequence of steps serves to build, deploy, and manage your AMI:

1. After getting an AWS account as described above, edit the settings.py file to include your AWS key and ID.

import os

# Add your Amazon AWS keys
AWS_ACCESS_KEY_ID = os.getenv('AWS_ACCESS_KEY_ID', 'your_id_here')
AWS_SECRET_ACCESS_KEY = os.getenv('AWS_SECRET_ACCESS_KEY', 'your_key_here')

#AMI_MASTER_TYPE = 'm1.large'
AMI_MASTER_TYPE = 't1.micro'
AMI_WORKER_TYPE = 't1.micro'

KEYPAIR_NAME = 'nimbus'
KEY_FILENAME = '%s.pem' % KEYPAIR_NAME

SECURITY_GROUP = 'nimbus'
PROJECT_NAME = 'demosite'

# Usually, do not need to change these
INSTANCE_PLACEMENT = 'us-east-1b'
BLANK_AMI_ID = 'ami-4a0df923'    # An empty Ubuntu AMI for us-east

2. Run fab make_master. This creates a new, running AMI starting with just the operating system and then installing the web stack. If needed, new ssh keys are generated and stored in nimbus.pem (other other file if specified in settings.py). If an Amazon security group does not yet exist, it is created (default group is “nimbus”, edit the settings.py file to change).

This initial install can take ten minutes to complete. At the end, the script displays the URL for the new running instance (and it stores that name locally in a system file named .nimbus).

3. At this point, the instance is populated with the whole web stack but does not have the demonstration application uploaded, deployed, and activated. Run fab deploy to complete the process and bring-up a live, sample website. This can take about five minutes.

4. To see the status at any time, run fab status which shows which instances are running or stopped, their DNS names, and Amazon instance identifiers.

(env) $ fab status
Instances:
master i-82610def running ec2-67-202-6-62.compute-1.amazonaws.com (67.202.6.62)
worker i-4e4a2423 running ec2-75-101-235-214.compute-1.amazonaws.com (75.101.235.214)
worker i-5a4b2537 running ec2-204-236-198-55.compute-1.amazonaws.com (204.236.198.55)

Open the AWS console and you will see something very close to this:

5. To see the live website, either type the DNS name in a web browser or run fab browse.

(env) $ fab browse

The demo application will be opened in the default browser:

Now, the application is running and world-visible. There are fab commands to stop, start, and restart and well as viewing the status.

Make edits to the sample web pages in demosite directory and upload them to a running instance with fab update.

When the website starts to become popular, manage scaling with fab distribute_load which will clone the running instance to make worker instances. The master instance directs traffic among the workers using Nginix. The master instance’s own webserver is still available as a fail-over, but the workers will get the first chance to handle load. The database and memcache continue to run on the master instance.

The full list of available options from the command line are:

browse           Open the master instance in the default browser
deploy           Install the application specified in settings.py to the...
distribute_load  Distribute load between as many workers as specified by...
fix_names        Reconfigure instances with the current domain names
make_master      Create a pre-configured instance
restart          Restart an instance
start            Start an instance
status           Lists instances
stop             Stop an instance
update           Update the installed application

Direct Access

Whether cloned from ActiveState’s prebuilt instance or built with a script, a running instance can be directly accessed using ssh (or putty on Windows). The default username is “ubuntu” but this can be changed in setting.py as long as you create a corresponding Amazon user account. The security is provided by the ssh2 key stored in nimbus.pem. For example:

ssh -i nimbus.pem ec2-50-16-18-225.compute-1.amazonaws.com

Upon logging-in, the Python virtual environment should already be active and show the (env) prompt. If not, start it with source /var/www/demosite/env/bin/activate. The virtual environment provides a Python setup that is fully independent of the one that ships with the operating system. This provides great flexibility in upgrading Python versions and controlling exactly which packages are installed. Activating the virtual environment makes all of the installed python packages visible to the command line interpreter.

For example, the memcache and database can be directly accessed from Python:

(env)ubuntu $
>>> import memcache
>>> mc = memcache.Client(['127.0.0.1:11211'], debug=0)
>>> mc.set("some_key", "Some value")
>>> value = mc.get("some_key")

>>> import MySQLdb, pprint
>>> cur = conn.cursor()
>>> conn = MySQLdb.connect(host='127.0.0.1', db='demosite', 
                           user='demosite', passwd='demosite')
>>> cur.execute('SELECT * FROM django_content_type')
9L
>>> pprint.pprint(cur.fetchall())
((1L, 'permission', 'auth', 'permission'),
 (2L, 'group', 'auth', 'group'),
 (3L, 'user', 'auth', 'user'),
     . . .

The Django configuration and control scripts are located in the /var/www/demosite directory:

demosite
	initial_data.json	Sample data for the database
	manage.py		Django file
	settings.py		Django settings file
	site.wsgi		Tell Apache run Django
	static			Directory of static content including CSS and JavaScript files
	templates		Directory for dynamic content

If you’re new Django, quick start tutorials can be found at http://code.djangoproject.com/wiki/Tutorials. And the Apache server log files are stored in the /var/log directory. If you’re new to Apache and need to modify the set-up, take a look at the Apache how-to guides at http://httpd.apache.org/docs/2.0/howto/.

You now have a running web server and all the tools needed to customize and manage the site:

Resources and Documentation:

ActivePython Business Edition includes our stable, supported, quality-assured Python distribution used by millions of developers around the world for easy Python installation and quality-assured code. Read more »
ActivePython Enterprise Edition
The Python Package Manager (PyPM) is the package manager for ActivePython. It provides quick installation of thousands of packages for many Python versions and platforms. Read more »