Tuesday, March 14, 2017

Introduction to PostGIS

Introduction to PostGIS


  • 1. Welcome
  • 2. Introduction
  • 3. Installation
  • 4. Creating a Spatial Database
  • 5. Loading spatial data
  • 6. About our data
  • 7. Simple SQL
  • 8. Simple SQL Exercises
  • 9. Geometries
  • 10. Geometry Exercises
  • 11. Spatial Relationships
  • 12. Spatial Relationships Exercises
  • 13. Spatial Joins
  • 14. Spatial Joins Exercises
  • 15. Spatial Indexing
  • 16. Projecting Data
  • 17. Projection Exercises
  • 18. Geography
  • 19. Geometry Constructing Functions
  • 20. More Spatial Joins
  • 21. Validity
  • 22. Equality
  • 23. Linear Referencing
  • 24. Dimensionally Extended 9-Intersection Model
  • 25. Clustering on Indices
  • 26. 3-D
  • 27. Nearest-Neighbour Searching
  • 28. Tracking Edit History using Triggers
  • 29. Advanced Geometry Constructions
  • 30. Tuning PostgreSQL for Spatial
  • 31. PostgreSQL Security
  • 32. PostgreSQL Schemas
  • 33. PostgreSQL Backup and Restore
  • 34. Software Upgrades
  • 35. Appendix A: PostGIS Functions
  • 36. Appendix B: Glossary
  • 37. Appendix C: License

INSTALLING POSTGRE SQL & POSTGIS

  1. Install Postgre SQL from the following location –  http://www.postgresql.org/download/windows/
  2. Install PostGIS from the following location http://download.osgeo.org/postgis/windows/pg94/
A useful tutorial for installing Postgre SQL & PostGIS can be found here but the process is pretty self-explanatory if you have some familiarity with databases and/or GIS.

CREATING YOUR FIRST POSTGIS SPATIAL DATABASE

  1. Open pgAdminIII from within the Postgre start menu folder
  2. Login to your PostgreSQL database by entering in the admin password you set on installation
  3. Create a new database by right clicking on databases and clicking new database – fill in the name (e.g. gisdb) and the owner then click ok
  4. Select the database that you created and add postgis as an extension so that it becomes a spatial database

LOADING SHAPE DATA INTO YOUR POSTGIS DATABASE

PostGIS comes with a very easy to use shape file importer which makes it trivial to populate your spatial database
  1. Open the PostGIS Shapefile Import/Export Manager
  2. Connect to your database by clicking on View Connection details and entering the username, password and database name
  3. Click Addfile and select the shapefile that you want to convert into a table

ACCESSING YOUR DATA USING QGIS

QGIS is a freely available Geographic Information System that may be used for manipulating and visualising spatial data. Its a tool I like to use for visualisation prior to taking my data in R. Using a spatial database rather than shapefiles allows me to interact with the data much more seemlessly than constantly exporting files between the two.
To view your data held in PostGIS in QGIS perform the following:
  1. Click Layer -> Add Layer -> PostGIS
  2. Create a New PostGIS connection
  3. Connect to the PostGIS connectino
  4. Select the data of interest
  5. Click Add
  6. Select Coordinate Reference System

ACCESSING YOUR DATA USING R

Accessing the data from within R is straight forward once you have developed an appreciation of SQL. (Unfortunately for Windows users it is not possible to seemlessly interact with the data using rgdal library due to driver compilation issues. However it can be accessed relatively seemlessly using RPostgreSQL library – many thanks to  for his neat post which shows a work around using readWKT.)
I provide the following function below to illustrate how to read a table from PostGIS and convert it to a SpatialPolygonsDataFrame. This code is based on Lee’s post but I’ve removed the for-loop with mapply to improve performance 🙂
library(RPostgreSQL)
library(rgeos)
library(sp)

dbname <- "gisdb"
host <- "127.0.0.1"
user <- "user"
password <- "password"
table <- "msoa_within_50kbristol"

getTable <- function ( table )
{
 # Load data from the PostGIS server
 conn = dbConnect(
 dbDriver("PostgreSQL"), dbname=dbname, host=host, port=5432, 
 user=user, password=password
 )
 
 strSQL = "
  SELECT ST_AsText(geom) AS wkt_geometry , *
  FROM msoa_within_50kbristol
  "
  
 dfQuery = dbGetQuery(conn, strSQL)
 dfQuery$geom <- NULL
 row.names(dfQuery) = dfQuery$gid
  
 # Create spatial polygons
 rWKT <- function (var1 , var2 ) { return (readWKT ( var1 , var2) @polygons) }
 spL <- mapply( rWKT , dfQuery$wkt_geometry ,  dfQuery$gid )
 spTemp <- SpatialPolygons( spL )
  
 # Create SpatialPolygonsDataFrame, drop WKT field from attributes
 spdf = SpatialPolygonsDataFrame(spTemp, dfQuery[-1])
 
 return (spdf)
}


bristol <- getTable( table )
plot(bristol)
head ( bristol@data )

LEARN HOW TO PERFORM SPATIAL QUERIES

A nice introduction to SQL and spatial SQL for Postgre can be found here but to get you started I’ve created some basic and hopefully  self-explanatory spatial queries to get you started.
Union of geometry containing the word Bristol in column msoa11nm
SELECT ST_Union(geom) FROM msoa_within_50kbristol WHERE msoa11nm LIKE '%Bristol%'
1k Buffer around geometry containing the word Bristol in column msoa11nm
SELECT ST_BUFFER(geom , 1000) FROM msoa_within_50kbristol 

SENDING DATA TO POSTGIS USING C#

It turns out that sending your data to PostGIS from C# is a straightforward task once you know how 🙂
Step 0
Download Npgsql using nuget – this can easily be done inside Visual Studio by right clicking on References and clicking on manage NUGet Packages.
Step 1
Establish a connection to your database. This is simple assuming you know your userid, password and database name.
  • Create a connection string detailing your userid, password and database
  • Create an NpgsqlConnection object
  • Open the connection
Step 2
  • Create an SQL command string e.g. INSERT INTO – if you are not familiar with SQL check out W3Schools
  • Create an NpgsqlCommand object based on the SQL command and the connection string
  • Execute the NpgsqlCommand
  • Close the connection
The code to perform this is written below in a method called InsertInTable:
static void InsertInTable()
        {
            try
            {

                // Making connection with Npgsql provider
                NpgsqlConnection conn = new NpgsqlConnection(connstring);
                conn.Open();
                // sql statement
                string unixtime, date, stime, lat, lon, creatorId, twitterId, text, coord;
                unixtime = "5645455";
                date = "11/09/2015";
                stime = "12:00:00";
                lat = "54.0";
                lon = "55.0";
                creatorId = "Terry";
                twitterId = "2131234523";
                text = "Happy times :-)";
                coord = String.Format("ST_MakePoint({0} , {1}, 4326)" , lat , lon);

                string sql = "INSERT INTO twittertest (unixtime, date, time , lat , lon , \"creatorId\" , \"twitterId\" , text , coord) " +
                                         String.Format( "Values ({0}, '{1}' , '{2}' , {3} , {4} , '{5}' , '{6}' , '{7}' , {8} )",
                                         unixtime,
                                         date,
                                         stime,
                                         lat,
                                         lon,
                                         creatorId,
                                         twitterId,
                                         text,
                                         coord);

                // Execute command
                NpgsqlCommand command = new NpgsqlCommand(sql, conn);
                Int32 rowsaffected = command.ExecuteNonQuery();

                // since we only showing the result we don't need connection anymore
                conn.Close();
            }
            
            catch (Exception msg)
            {
                // something went wrong, and you wanna know why
                Console.WriteLine(msg.ToString());

            }
        }
The eagle eyed may have noticed that  to insert a point within the database a non-standard SQL command is used. A geometry constructor called “ST_MakePoint” is used to create the point – you can find the documentation for it here.

RETRIEVING DATA FROM POSTGIS USING C#

Step 1
Establish a connection to your database. This is simple assuming you know your userid, password and database name.
  • Create a connection string
  • Create an NpgsqlConnection object
  • Open the connection
Step 2
  • Create an SQL command string e.g. SELECT * FROM Tablename
  • Create an NpgsqlCommand object based on the SQL command and the connection string
  • Execute the NpgsqlCommand
  • Close the connection
The code to perform this is in the method below called GetTable:
static DataTable GetTable()
        {
            DataSet ds = new DataSet();
            DataTable dt = new DataTable();

            try
            {
            
                // Making connection with Npgsql provider
                NpgsqlConnection conn = new NpgsqlConnection(connstring);
                conn.Open();
                // sql statement
                string sql = "SELECT unixtime , \"twitterId\" , CAST ( coord AS text) FROM twittertest";

                NpgsqlCommand cmd = new NpgsqlCommand(sql);
                cmd.AllResultTypesAreUnknown = true;

                // data adapter making request from our connection
                NpgsqlDataAdapter da = new NpgsqlDataAdapter(sql, conn);

                // since we only showing the result we don't need connection anymore
                conn.Close();

                ds.Reset();
                da.Fill(ds); // filling DataSet with result from NpgsqlDataAdapter
                dt = ds.Tables[0]; // since it C# DataSet can handle multiple tables, we will select first
                return dt;
            }
            catch (Exception msg)
            {
                // something went wrong, and you wanna know why
                Console.WriteLine(msg.ToString());
                return (dt);
            }
        }
You may have noted that I cast the the coord (spatial object) as text so that it can be read into C# as a string. A useful alternative is to use the ST_AsText command to convert the coord into a WKT formatted string – see the documentation here.
The full code to create a sample application is supplied below as an example for completeness.
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;
using System.Data;

using Npgsql;

namespace SQLPostGIS
{
    class Program
    {
        // PostgeSQL-style connection string
        static string connstring = String.Format("Server={0};Port={1};" +
            "User Id={2};Password={3};Database={4};",
            "localhost", "5432", "postgres",
            "password", "gisdb");

        static void InsertInTable()
        {
            try
            {

                // Making connection with Npgsql provider
                NpgsqlConnection conn = new NpgsqlConnection(connstring);
                conn.Open();
                // sql statement
                string unixtime, date, stime, lat, lon, creatorId, twitterId, text, coord;
                unixtime = "5645455";
                date = "11/09/2015";
                stime = "12:00:00";
                lat = "54.0";
                lon = "55.0";
                creatorId = "Terry";
                twitterId = "2131234523";
                text = "Happy times :-)";
                coord = String.Format("ST_MakePoint({0} , {1}, 4326)" , lat , lon);

                string sql = "INSERT INTO twittertest (unixtime, date, time , lat , lon , \"creatorId\" , \"twitterId\" , text , coord) " +
                                         String.Format( "Values ({0}, '{1}' , '{2}' , {3} , {4} , '{5}' , '{6}' , '{7}' , {8} )",
                                         unixtime,
                                         date,
                                         stime,
                                         lat,
                                         lon,
                                         creatorId,
                                         twitterId,
                                         text,
                                         coord);

                // Execute command
                NpgsqlCommand command = new NpgsqlCommand(sql, conn);
                Int32 rowsaffected = command.ExecuteNonQuery();

                // since we only showing the result we don't need connection anymore
                conn.Close();
            }
            
            catch (Exception msg)
            {
                // something went wrong, and you wanna know why
                Console.WriteLine(msg.ToString());

            }
        }

        
        static DataTable GetTable()
        {
            DataSet ds = new DataSet();
            DataTable dt = new DataTable();

            try
            {
            
                // Making connection with Npgsql provider
                NpgsqlConnection conn = new NpgsqlConnection(connstring);
                conn.Open();
                // sql statement
                string sql = "SELECT unixtime , \"twitterId\" , CAST ( coord AS text) FROM twittertest";

                NpgsqlCommand cmd = new NpgsqlCommand(sql);
                cmd.AllResultTypesAreUnknown = true;

                // data adapter making request from our connection
                NpgsqlDataAdapter da = new NpgsqlDataAdapter(sql, conn);

                // since we only showing the result we don't need connection anymore
                conn.Close();

                ds.Reset();
                da.Fill(ds); // filling DataSet with result from NpgsqlDataAdapter
                dt = ds.Tables[0]; // since it C# DataSet can handle multiple tables, we will select first
                return dt;
            }
            catch (Exception msg)
            {
                // something went wrong, and you wanna know why
                Console.WriteLine(msg.ToString());
                return (dt);
            }
        }

        static void Main(string[] args)
        {
            DataSet ds = new DataSet();
            DataTable dt = new DataTable();

            

            try
            {
                InsertInTable();
                dt = GetTable();
                foreach (DataRow row in dt.Rows)
                {
                    Console.WriteLine(row["unixtime"].ToString() + "," + row["twitterID"].ToString());
                }
                
                
            }
            catch (Exception msg)
            {
                // something went wrong, and you wanna know why
                Console.WriteLine(msg.ToString()); 
            }

            Console.ReadLine();
        }
    }
}

Sunday, March 12, 2017

Windows installers

Interactive installer by EnterpriseDB

Download the installer certified by EnterpriseDB for all supported PostgreSQL versions.
This installer includes the PostgreSQL server, pgAdmin; a graphical tool for managing and developing your databases, and StackBuilder; a package manager that can be used to download and install additional PostgreSQL tools and drivers. Stackbuilder includes management, integration, migration, replication, geospatial, connectors and other tools.
This installer can run in graphical or silent install modes.
The installer is designed to be a straightforward, fast way to get up and running with PostgreSQL on Windows.
Advanced users can also download a zip archive of the binaries, without the installer. This download is intended for users who wish to include PostgreSQL as part of another application installer.

Graphical installer by BigSQL

Download the graphical installer from BigSQL for all supported versions.
This distribution includes the PostgreSQL server, a graphical component manager, command line and graphical tools for managing databases, plus many open source community components.
Integrated components include web and desktop developer tools, geospatial, provisioning & management, compatibility & migration, backup/restore, integration with external databases (Cassandra, Oracle, SQL Server, Hadoop), and procedural languages (Python, Perl, Java, and TCL).
This distribution is a fast, developer-friendly way to get a complete PostgreSQL environment installed and running. It uses an open source toolchain to build PostgreSQL and extensions, which simplifies cross-platform development of extensions.
Advanced users can also download a command line version of the distribution, for scriptable installs or embedding with other applications.

C# connect postgresql database

Introduction

In this article, I would like to show you the basics of using a PostgreSQL database in your .NET application. The reason why I'm doing this is the lack of PostgreSQL articles on CodeProject despite the fact that it is a very good RDBMS. I have used PostgreSQL back in the days when PHP was my main programming language, and I thought.... well, why not use it in my C# application.

Background

I assume you have a basic knowledge of databases, SQL commands, and so on.

Why PostgreSQL

First and foremost, it's free*, in BSD-style freedom (both libre and gratis), which means you will not have license problems in your commercial applications. Next, you have a very good procedural language called PL/pgSQL (there are also the Java, Perl, Python, Tcl (and other) flavors). Lots of communities support it (large ones like Planet PostgreSQL, or blogs, for instance, "Depesz" blog), and finally, lot's of fun exploring this pearl of open source databases.

Installing PostgreSQL and the .NET data provider

First of all, go to the PostgreSQL website. PostgreSQL is known for having a very good documentation system. For the purposes of this article, I used PostgreSQL v.8.3.4 and PgAdmin III v. 1.8.4 (that comes bundled with PostgreSQL). Remember to install the database and PgAdmin III, but NOT Npgsql, since it comes with version 1.x and we're going to use v. 2.0. from a different package. Now, if you have a PostgreSQL server up and running (which is a really easy task), let's download some drivers. For this article, I have used NpgSQL2 built directly for MS .NET. When you download it, in the zip package (\bin directory), you will find Npgsql.dll, and that's the library we're looking for.

Playing with PgAdmin III

If you have ever worked with any DB-Management software like Aqua Data StudioSQL Manager (I use the Lite version every day), or the MS Management Studio Express, you will find PgAdmin III really intuitive. It may not have that many gizmos as others, but it will do the job nicely. Here are a couple of daily tasks that you probably want to know how to do.
  1. Creating a new user (unless you really don't mind playing with a super-user account :-)):
  2. How to create new role on the server
  3. Creating a new database (fairly obvious, but nevertheless here it goes :-)):
  4. How to create new database in PgAdmin III
  5. Creating a table:
  6. How to create new table in PostgreSQL
  7. Adding stuff to tables (there are, of course, other ways, but I will let you explore this):
  8. How to add more tables
  9. If you are looking for an MS-SQL Identity, just make your column type serial, or look for sequence in the documentation. OK, to be completely honest, PgAdmin III still needs work... numbers may not appear if you add a row through the grid, but believe me, they are there :-). You will see them the next time you reload a window with the grid.
  10. If you looking for more hit documentation :-)

Making a database

Here is a SQL-script so we can make our test database. Remember to create a database and a user. My user for this article will be adrian.
CREATE TABLE simple_table
(
  id integer NOT NULL, -- SERIAL if like identity-like functionality
  tekst character varying(50),
  CONSTRAINT simple_table_pkey PRIMARY KEY (id)
)
WITH (OIDS=FALSE);
ALTER TABLE simple_table OWNER TO adrian; -- my user

-- i drop Polish letters so i don't have to worry about codepage
INSERT INTO simple_table (id, tekst) VALUES (1, 'jaki tekst');
INSERT INTO simple_table (id, tekst) VALUES (2, 'jaki inny tekst');
INSERT INTO simple_table (id, tekst) VALUES (3, 'jeszcze inny tekst');

Using PostgreSQL in C#

Just make an empty Windows Forms application, and add a reference to Npgsql.dll, like shown in the picture:
Adding Npgsql provider

Baking a connection string PostgreSQL-style

A connection string in PostgreSQL is made by making keyword=value; pairs (Important: even the last element must end with a ';'). The most basic ones are:
  • Server - specifies the server location
  • User Id - the database user
  • Port - default is 5432
  • Password - the password for the database user
  • Database - the database name

Simple test application

using Npgsql;

namespace PostgreSQLTEst
{
    public partial class Form1 : Form
    {
        private DataSet ds = new DataSet();
        private DataTable dt = new DataTable();
        public Form1()
        {    
            InitializeComponent();    
        }
        private void llOpenConnAndSelect_LinkClicked(object sender, 
                     LinkLabelLinkClickedEventArgs e)
        {
            try
            {
                // PostgeSQL-style connection string
                string connstring = String.Format("Server={0};Port={1};" + 
                    "User Id={2};Password={3};Database={4};",
                    tbHost.Text, tbPort.Text, tbUser.Text, 
                    tbPass.Text, tbDataBaseName.Text );
                // Making connection with Npgsql provider
                NpgsqlConnection conn = new NpgsqlConnection(connstring);
                conn.Open();
                // quite complex sql statement
                string sql = "SELECT * FROM simple_table";
                // data adapter making request from our connection
                NpgsqlDataAdapter da = new NpgsqlDataAdapter(sql, conn);
                // i always reset DataSet before i do
                // something with it.... i don't know why :-)
                ds.Reset();
                // filling DataSet with result from NpgsqlDataAdapter
                da.Fill(ds);
                // since it C# DataSet can handle multiple tables, we will select first
                dt = ds.Tables[0];
                // connect grid to DataTable
                dataGridView1.DataSource = dt;
                // since we only showing the result we don't need connection anymore
                conn.Close();
            }
            catch (Exception msg)
            {
                // something went wrong, and you wanna know why
                MessageBox.Show(msg.ToString());
                throw;
            }
        }
    }
}
And the effect:
Working example
And, that's it! :D. Practice PostgreSQL and enjoy!