Installation
============
Different versions can be downloaded from `www.simo-project.org/download/releases/latest/ `_.
Executable version
------------------
If you downloaded the executable distribution and decompressed the downloaded file, you've all set up. You can jump directly to :ref:`usage` section.
IMPORTANT: If you are using the executable distribution on Windows, you MUST install the `Visual C++ 2008 Redistributable package `_ or you will get cryptic errors on missing dlls or incorrect Side-by-side configuration.
Source version
--------------
The source distribution needs two more step besides decompression before it's usable.
You need to have Python programming language installed. You can get it from `www.python.org `_. SIMO is not ready for Python 3, so download the latest Python 2.6.x version. Installing Python is usually necessary only on Windows, on Linux and OS X you should already have a working Python installation.
Besides Python, you'll need a Python package called setuptools. You can test whether you have setuptools installed by writing *easy_install* on the command line. If the output is something like "error: No urls, filenames, or requirements specified (see --help)" you're good to go, but if easy_install is not recognized as a command, download `ez_setup.py `_ and run::
python ez_setup.py
After you've got working versions of Python and setuptools, the next steps depend on whether you're using a Unix variant or Windows.
Linux and OS X users
********************
On the command line in the directory into which you decompressed the downloaded SIMO source distribution, issue this command::
python build_env.py
Then go and grab a cup of coffee, the building process will take a good while. When it finishes you should have a working SIMO installation and you are ready for the :ref:`usage` section.
If you run into trouble in the *python build_env.py* stage, have a look at the :ref:`known_issues` section.
Building a 32 & 64 bit Python on OS X 10.6 Snow Leopard
+++++++++++++++++++++++++++++++++++++++++++++++++++++++
Buildout will build a Python for you within the buildout environment, but in addition to that you might want to build a general Python version that is more up to date than the system Python that Apple provides. These instructions are based on `this blog post`.
Making sure you have 32- and 64-bit enabled readline. First download the latest readline release and then::
export MACOSX_DEPLOYMENT_TARGET=10.6
export CFLAGS="-arch i386 -arch x86_64"
export LDFLAGS="-Wall -arch i386 -arch x86_64"
./configure
make
sudo make install
Download latest Python source release and::
./configure --enable-framework --with-universal-archs=intel \
--enable-universalsdk=/Developer/SDKs/MacOSX10.6.sdk
Now you should have a working Python installation in /Library/Frameworks/Python.framework
Windows users
*************
Python 2.6 (recommended):
+++++++++++++++++++++++++
1. Download and install `Python 2.6 `_ (if you didn't already)
2. Download and install the compatible C compiler: `Visual C++ 2008 Express `_
3. Install setuptools, which also includes easy_install, using the exe-installer found `here `_
4. Download `sqlite3 amalgamation source `_
5. Download `pysqlite source (.tar.gz): `_
6. Download and install something that can unpack tar and gzip files, (e.g. `7-zip `_
7. Unpack sqlite3 to a location of your choosing, start up the Visual Studio 2008 command prompt (under tools in the visual studio start menu), navigate to where you unpacked sqlite3 and run the following (two separate commands)::
cl /c /O2 /D SQLITE_ENABLE_LOAD_EXTENSION /D SQLITE_ENABLE_RTREE /D SQLITE_THREADSAFE /D HAVE_READLINE sqlite3.c
link /LIB /OUT:sqlite3.lib sqlite3.obj
8. Unpack pysqlite to a location of your choosing, find the setup.cfg in the folder and alter it to match the following (where is where you just compiled sqlite3)::
[build_ext]
include_dirs=
library_dirs=
libraries=sqlite3
define=HAVE_LOAD_EXTENSION,SQLITE_ENABLE_RTREE
9. Now run (in the folder where you extracted the source)::
python setup.py build --force
python setup.py install
Python 2.6+ (IF compiling fortran libraries):
+++++++++++++++++++++++++++++++++++++++++++++
1. Download and install the compatible C compiler: `MinGW `_ (make sure it's on PATH for simplicity)
2. In the mingw installation folder, copy lib\libmingw32.a -> \libs\mingw32.lib and lib\libmingwex.a -> \libs\mingwex.lib
3. In \libs run the following::
lib -remove:mbrtowc.o mingwex.lib
lib -remove:wcrtomb.o mingwex.lib
Python 2.5:
+++++++++++
1. Download and install `Python 2.5 `_ (if you didn't already)
2. Download and install the compatible C compiler: `MinGW `_ (make sure it's on PATH for simplicity)
3. Install setuptools, which also includes easy_install, using the exe-installer found `here `_
4. Download `sqlite3 amalgamation source `_
5. Download and install something that can unpack tar and gzip files, (e.g. `7-zip `_
6. Unpack sqlite3 to a location of your choosing, navigate to that location in command prompt and run *with the mingw gcc*::
gcc -O2 -USQLITE_OMIT_LOAD_EXTENSION -DSQLITE_ENABLE_LOAD_EXTENSION=1 -DSQLITE_ENABLE_RTREE=1 -DSQLITE_THREADSAFE=2 -DHAVE_READLINE=1 -c sqlite3.c
gcc -shared -s -o sqlite3.dll *.o
7. Copy sqlite3.dll to C:\WINDOWS\SYSTEM32 (or to a folder in the path if you don't have access to system32). If you plan on using pyinstaller, you also need to copy sqlite3.dll to \DLLs
8. Download `pysqlite source `_
9. Unpack pysqlite to a location of your choosing, find the setup.cfg in the folder and alter it to match the following (where is where you just compiled sqlite3)::
[build_ext]
include_dirs=
library_dirs=
libraries=sqlite3
define=HAVE_LOAD_EXTENSION,SQLITE_ENABLE_RTREE
Now run (in the folder where you extracted the source)::
python setup.py build --force -c mingw32 install --force
Common:
+++++++
10. Download and run the PostgreSQL `one click installer `_ when asked if you would like to install additional packages, choose yes and install PostGIS (this can also be done afterwards with the included StackBuilder, but it's easier to do here) and let it make the template/example when it asks.
11. Download and install the latest version of `psyocopg2 `_
12. Download and install `lxml `_ latest exe-installer for your system/python
13. Download and install `numpy `_ exe-installer for your system/python.
14. Make sure you have the easy_install script in your system path (or go to the folder where it is) and execute the following in the command prompt
::
easy_install -Z cython (on 2.6 run in the Visual Studio 2008 command prompt)
easy_install -Z ZODB3
easy_install -Z pygooglechart
easy_install -Z python-dateutil
easy_install -Z pyparsing
If easy_installing cython fails, you can get ready made `Windows binaries `_
If easy_installing pyparsing, dry specifying the pyparsing package explicitly
::
easy_install http://cheeseshop.python.org/packages/source/p/pyparsing/pyparsing-1.5.5.tar.gz
16. If you want to run tests, you'll also need
::
easy_install nose
easy_install minimock
easy_install interlude
17. If you want to run code profiling (only for developers)
::
easy_install guppy
18. If any of the above easy_installs failed, see if there's an exe-installer available for your system and use that instead (or you can do it anyway, it shouldn't make any difference).
19. Download libiconv, geos, proj and libspatialite from the spatialite install `site `_
20. Unzip the zips from the previous step and copy all dll-files from the bin folders into C:\WINDOWS\SYSTEM32 and/or to \src\simo\db\datadb
21. Now you are ready to build a working SIMO installation. On the command prompt, while in the directory where you unzipped the source distribution, run
::
python build_env.py
Enabling parallel execution
+++++++++++++++++++++++++++
If you wish to run simo in parallel, you will also need to do the following:
1. Download `redis `_, unzip and copy the files to a location of your choosing (for example C:\redis, adding it to the path is convenient for usage)
2. run easy_install -Z Celery
3. run easy_install -Z redis
SVN Checkout
------------
There's a third option for installing SIMO if you want to track the latest and greatest developments. You can do a `subversion `_ checkout for the SIMO codebase and you can track the development day in day out. The checkout address is::
svn.simo-project.org/simo/trunk
After the checkout you essentially have the source distribution (plus a few bits and pieces), so follow the instructions above.
Optimizing PostgreSQL
+++++++++++++++++++++
For Unix, change /etc/sysctl.conf (PLEASE backup this file first!)
Linux: kernel.shmmax=value
Mac OS: kern.sysv.shmmax=value
FreeBSD: kern.ipc.shmmax=value
Note, if getconf is on your system, you can run the following to get good inital values for shmmax and shmall:
#!/bin/bash
# simple shmsetup script
page_size=`getconf PAGE_SIZE`
phys_pages=`getconf _PHYS_PAGES`
shmall=`expr $phys_pages / 2`
shmmax=`expr $shmall \* $page_size`
echo kernel.shmmax = $shmmax
echo kernel.shmall = $shmall
sysctl.conf (on a 4 core 16GB RAM system)
kernel.shmall = 2057198
kernel.shmmax = 8426285056 # note, this value must be equal or greater than postgresql shared_buffers setting, though roughly half of your RAM is a good example.
kernel.sem = 250 32000 100 128 # may need to increase these further if a large number of processes are on the same system
# Potentially helpful, but untested settings
# vm.swappiness=0 # try to avoid swapping; may hurt performance if you run out of memory
# vm.overcommit_memory=2 # don't let processes allocate more memory than they need.
Also, consider setting 'noatime' to the database volume mount options to disable constantly updating file access times.
reboot (or sudo sysctl -p should also work)
Use pgtune (http://pgfoundry.org/projects/pgtune/), which on a 4 core 16GB RAM system resulted in the following (IMPORTANT! note the non pgtune wizard entries):
#------------------------------------------------------------------------------
# CUSTOMIZED OPTIONS
#------------------------------------------------------------------------------
# SSD options (only if you have a seperate ssd)
data_directory = '/ssd/postgresql_data' # use data in another directory
seq_page_cost = 0.05
random_page_cost = 0.1
#custom_variable_classes = '' # list of custom variable class names
default_statistics_target = 50 # pgtune wizard 2011-10-12
maintenance_work_mem = 960MB # pgtune wizard 2011-10-12
constraint_exclusion = on # pgtune wizard 2011-10-12
checkpoint_completion_target = 0.9 # pgtune wizard 2011-10-12
effective_cache_size = 11GB # pgtune wizard 2011-10-12
work_mem = 96MB # pgtune wizard 2011-10-12
work_mem = 128MB # book recommendation for speed
wal_buffers = 8MB # pgtune wizard 2011-10-12
wal_buffers = 16MB # book recommendation for any server (but especially on heavy insert work)
checkpoint_segments = 16 # pgtune wizard 2011-10-12
shared_buffers = 3840MB # pgtune wizard 2011-10-12
max_connections = 80 # pgtune wizard 2011-10-12
# BULK LOAD OPTIONS, DISABLE AFTER DONE!!!
checkpoint_segments = 128 # or even 256?
checkpoint_timeout = 1800 # in seconds; 30 min
work_mem = 256MB # improves order by, distinct etc
# synchronous_commit = off # this can be done clientside, so no need to do it here. Iit will mean however that the data may be malformed on catastrophe
# fsync = off # may be unnecessary and could destroy the existing database if something goes wrong!!!