Blog

Silicon Valley Postgres Meetup: How to Auto-cache Postgres with no code changes

The first meeting of the Silicon Valley Postgres Meetup was last night. Amazon Web Services sponsored the facilities in Cupertino and Roland Lee from Hemdalldata presented on:

How to Auto-Cache Postgres with no code changes.

Roland Lee

There were about 20 people in attendance as well as another half a dozen that participated via Amazon Chime. Debbie Cerda, our Director of Business Development flew out from Austin, Tx to host. When we launched the Silicon Valley Meetup we wanted to ensure that it would not conflict with the well respected San Francisco PUG. Based on initial response, there is not a conflict and we are very happy about that. We attribute the lack of conflict two items:

  1. San Francisco is not part ...

Read More

Postgres at Seattle Web Developers Meetup, recap

Postgres: The center of your data universe

Postgres at Seatte Web Developers

This talk is proving to be great content for those who are not necessarily Postgres Users. Last night I presented this talk at the Seattle Web Developers Meetup. The location was Adobe, next to Google and Tableau. I didn't even know there was a small tech complex on N. 34th in Seattle. There were about 27 people, which falls in line with the 50% rule of Meetups[1]. Here are my observations from the audience:

  1. Surprisingly the majority of developers were Python web developers (as a group)
  2. Many of them do not use Postgres and wanted to hear more about it
  3. There were several government employees that were wondering how to get ...

Read More

Postgres, upcoming community awesomeness

Upcoming community awesomeness

Now that summer is over and we have officially decided never to schedule anything in August again, we need to share a bunch of upcoming community goodness!

Goals

Never lose sight of the goal

 

  • PGConf US and NYCPUG are hosting: PGConf US Mini: NYC tomorrow! Bruce Momjian will be speaking on Postgres v10. This is the second mini that PGConf US has organized. The first was last May in Austin. It looks like a great lineup and turnout.

 

  • I ...

Read More

Postgres Load Testing with pgreplay

Overview

Usually we want to test before deploying changes to a production Postgresql cluster. Commonly the test itself is executed in a context that is not very similar to the production environment. How can you run a test that is realistic so there are no horrible surprises when you deploy on the production system?  Read on! This document describes a procedure where changes can be tested on a system that is running a load that is nearly identical to a production system.

This procedure utilizes pgreplay, which reads postgres' server logs and executes the sql statements found there, with the same timing that they were executed on the production system.

This procedure also uses lvm and its volume snapshot ...

Read More

Postgres v10: An Amplified version of PostgreSQL at VanDev and Vancouver Postgres tonight!

If you are looking for all the skinny on Postgres v10, I have just the meetup for you. I will be giving my presentation: Postgres V10: An Amplified version of PostgreSQL tonight at a joint meeting of Vancouver Developers Network and Vancouver Postgres Meetup. Be there or be square.

In this presentation I go over all the major features in Postgres v10 including but not limited to:

  1. BigData
  2. Replication and Scaling
  3. Administration
  4. SQL Features
  5. XML and JSON

Join the meetups, its fun and will make you a better human.

Read More

Announce: Denver Postgres User Group

After much deliberation with the CMD community team we have launched the Denver Postgres User Group! We hope that our community in Denver, Boulder, and Colorado Springs will join us at upcoming events and submit content. It has been a long time since we have had an active Denver group and Denver is a hot bed for Postgres external development. Our first meeting will be announced soon and should be expected in October. Have ideas on facilities or content? Please contact us on the meetup page. We would love to see 3 - 4 meetings before PGConf US National!

Read More

Postgres deferred PRIMARY KEYS, a hidden gem

Oracle 7.3 supports it!

That is how this all started. A gentleman tweeted about a Postgres limitation that Oracle has not had since at least since Oracle 7.3

The problem

As you can see in the tweet, Postgres by default will not defer a PRIMARY KEY check. Without the check being deferred the following will not work:

postgres=# select * from demo; 
id  
----
 1
 2
(2 rows)
postgres=# alter table demo add primary key(id);   
ALTER TABLE
postgres=# begin;
BEGIN
postgres=# update demo set id=id-1;
UPDATE 2
postgres=# update demo set id=id+1;
ERROR:  duplicate key value violates unique constraint "demo_pkey"
DETAIL:  Key (id)=(1) already exists.

  The solution

The solution, as mentioned ...

Read More

Postgres autovacuum, bloat and tpc-c style workloads

For most workloads the Postgres Autovacuum daemon works just fine. You go about your day with 3 workers that wake up once a minute to make sure that everything is nice and tidy. If things are dirty enough (around 10%) then one of the workers gets in gear and cleans things up. Unfortunately, if you have an inverted load from the norm, Autovacuum may not be able to keep up and you will suffer increased fragmentation and bloat.

The norm

The most common workload for Postgres (especially web based apps) includes many reads and some writes. It is usually somewhere around 75-95% reads and 5-25% writes. Assuming an average database, Autovacuum will keep up just fine and with tuning will ...

Read More

Where is the Postgres community?

A recent poll was conducted @amplifypostgres to determine where the Postgres community should have its interactive communication. Options included were Google Hangouts, Slack, Reddit or “Other”.

The results were not surprising, with Google Hangouts beating Slack with 157 votes cast. There were also notable mentions of IRC, and Gitter. A couple of long time Postgresql.Org members asked the inevitable, “What is wrong with IRC?” Of course there is nothing wrong with IRC but when you tell many community users to use IRC, the most common response is “IRWhat?” which is either a sign of disdain or ignorance depending on the user.

The problem and what is driving this post was an additional comment made by a long time community ...

Read More

Why Postgres? (How did I get here?)

You may ask yourself, how did I get here?

The journey to this place in my professional career as the newly hired Director of Business Development for Command Prompt, Inc. is a long and winding one, and so i'd like to share a couple of stories to enlighten curious folks:

“Why Postgres?!”

Last September the fledgling consulting firm that I was handling sales and business development for was shuttering, and my best bud and business colleague Jim Nasby and I were on the market for job opportunities. Word had gotten out, and I had an introductory call with a potential employer. An advantage of my former position at Blue Treble was the ability to engage regularly in the ...

Read More

PostgreSQL: The Center of your Data Universe @ Datalayer Conference

You should always be careful of what you ask for. A couple of months ago while I was feeling particularly brave, I submitted to present at the DataLayer Conference. The next thing I knew, I was speaking at the DataLayer Conference. The conference takes place in Austin on May 17th. Conferences like this one are an awesome channel for PostgreSQL advocacy. Of course I am partial to PgConf US but a conference such as DataLayer allows us to reach people who may be using one of those "other" databases. 

Here is the synopsis of the presentation I will be giving:

PostgreSQL: The Center of Your Data Universe

Although there are still battles to be fought, the war has already ...

Read More

Indirect Advocacy

Last week I spoke at the Bellingham Young Professional Group on starting and running a business. It was a well attended presentation. I was nervous at first because although I do a lot of public speaking, I usually speak to technical people. This was a wholly different crowd and I was pulling from a different set of expertise. The crowd was largely under 30 and wanting to start a business of some sort. The presentation went over well and by far the most common feedback was, "I didn't even consider that, thank you". It is a good feeling to know you are helping people.

Indirect Advocacy: not directly caused by a recommendation of a particular cause

By stepping out ...

Read More

PostgreSQL for Oracle people

Below is the video of the webinar I did recently on PostgreSQL and Oracle. This webinar went very well. This is the first time I had ever performed a webinar that I recall. It was an interesting experience.  

PostgreSQL for Oracle Developers and DBA's

 

If you would like more information on this topic or any other topic surrounding PostgreSQL and Open Source, don't hesitate to contact us.

Read More

Do not buy the closed source lie of free videos

There is a nice lie out there. A lot of people want to believe it. They think by believing this lie it will somehow increase something for them. In some ways that is true. If you want what you are doing to be about you. If you are a believer in Open Source, it isn't about you. It is about the community and bettering that community as a whole.

If we provide the videos of our sessions for free, you won't attend the conference.

The PgConf US conference grows every year and guess what, they provide their videos for free.

If you pay for the conference, we will provide the videos for free.

LinuxFest Northwest which is larger ...

Read More

What should I submit to PgConf US 2017?

From the title, that is the question. This is the last week of the PgConf US 2017 CFP (you can submit here: http://www.pgconf.us/2017/submit/) and I have no idea what to submit.

I am blessed that my talks are very well attended, the audience is engaged and we all have a good time. Many times laughing at me because I have a hard time staying on one specific topic (especially if someone brings a kid into the room). There is the disclaimer I have to put up on my slides because there are some in the community that can't handle humor or PG-13 content but we must all love our neighbor and enjoy them ...

Read More

Install LAPP in Containers

Install LAPP in Containers
(Linux Containers and Linux, Apache, PostgreSQL, PHP)


In this blog post I will detail how to install Apache, PHP, PostgreSQL in Linux containers on Ubuntu 16.04 LTS.


It can be desirable to isolate certain software from the rest of a system for a variety of reasons. These reasons can range, but one of the most common is security. There are a multitude of methods for isolation ranging from process sandboxing to full hardware virtualization. Regarding the former, we have a tool called chroot.


Change root, or chroot for short, has been a UNIX utility for “sandboxing” processes since 1982. Sandboxing is a general term to describe the act of executing processes outside of the root ...

Read More

Snap packages for 9.3.15, 9.4.10, 9.5.5 and 9.6.1 available

The snap packages for 9.3.15, 9.4.10, 9.5.5 and 9.6.1 are now available. To install them:

sudo snap install postgresql$version

Where $version is one of 93, 94, 95 or 96.

The snap packages for PostgreSQL are a community project being lead by Command Prompt. You can visit the repo at github.

To learn more about snap packages please visit the Ubuntu snapcraft developer FAQ.

Read More

Can I make initdb quiet?

A #postgresql user today asked:

noob question - trying to write a Dockerfile that runs postgres... how do I get the effect of a non-interactive `service postgresql initdb` call?

  While several other community members provided the truthful but not helpful answer of, "Just throw Docker in the Trash", I worked out the following hack/trick/snipe hunt. The answer is, you can't. You have to call initdb directly. This took a few tries because PostgreSQL does not ship -q (quiet) flag with initdb. It will always make noise even when you don't want it to. However, if you call initdb directly, pass a few flags that have nothing to do with actually being quiet and redirect STDERR then ...

Read More

psql tips: Change the location and filtering of the history file

Anyone who uses PostgreSQL knows of the best client available: psql. This is the client that ships with PostgreSQL. Yes it is a command line client (which turns some people off) but that also means that it is the most efficient at everyday tasks for a DBA. What a lot of people don't know is that psql is rather configurable. Here is an example:

Problem 1: I want my history file in a place other that ~/.psql_history

Problem 2: I want my history file to be per database not global

Solution 1: Edit the .psqlrc file and change the history file settings

\set HISTFILE ~/psql_history/.psql_history

This will put your .psql_history file into the directory psql_history under your home ...

Read More

Will Postgres use the second element of an index if it is the only element in the WHERE clause?

This is a test table from an Oracle to Postgres migration. The table has had a dozen or so columns removed for the illustration of this test case. I did not design this table but the customer is fixing it (adding proper primary key, changing to boolean and integer where appropriate etc...).

Table "public.costcenter"

       Column       |            Type             |     Modifiers
-------------------+-----------------------------+--------------------
  costcenterid      | numeric                     | not null
  costcenterno      | character varying(100)      | not null
  amount            | numeric(18,2)               |
  closed            | boolean                     | not null
  enteruser         | integer                     |
  phonelines        | numeric                     |
  cckeyid           | character varying(100)      |
  country           | character varying(2)        |
  level4            | character varying(100)      |
  level5            | character varying(100)      |
  level6            | character varying(100)      |
  level7            | character varying(100)      |
  level8            | character varying(100)      |
  level9            | character varying(100)      |
  latitude          | character varying(20)       |
  longitude         | character varying(20 ...

Read More

Rich in the Jungle: A AWS to Softlayer comparison for PostgreSQL

I have updated my Rich in the Jungle presentation with new pricing for AWS vs. Softlayer. Things haven't changed much, in terms of raw performance per dollar (which is not the only qualifier). Softlayer is clearly the winner.

Read More

The fall of Open Source

Once upon a time FOSS was about Freedom. It was about exposing equality within source code. It allowed everyone equal rights and equal access to the technology they were using. An idea that if you were capable, you could fix code or pay someone to fix code. An ideology that there was something greater than yourself and that there was an inherent right built into what it is to be human with software.

Leaders to lemmings

I sat in a bar slowly nursing beers with other community members over a period of hours. We spoke of many things. We spoke of the never-done new PostgreSQL website. We spoke of my distaste for Amazon Web Services since reformed, with the exception ...

Read More

Upgrading Ubuntu LTS and PostgreSQL

If your PostgreSQL instance is running on an Ubuntu LTS system that you need to upgrade to the most recent release, say from precise to trusty – because, well, sooner or later you must – you need to consider what is going to happen to your database.

Note
The upgrade process described in this article is similar to what you would have to do if you were upgrading from Trusty to Xenial, the newest Ubuntu LTS release.

Ubuntu attempts to make the process of upgrading to the newest distribution release easy and hassle-free. In fact, it is the case in many situations but not when there is PostgreSQL running in the system. If you just go ahead and try to run ...

Read More

What is good for the community is good for the company (profit is the reward)

As the PostgreSQL community continues down its path of world domination I can't help but wonder whether the various PostgreSQL companies are going to survive the changes. Once upon a time there was an undercurrent of understanding that what was good for the community was good for the company. Whatever company that may be. However, over the last few years it seems that has changed. It seems there is more prevalance toward: What is good for the company is good for the community, or in other words, "The goal is profit."

That is a flawed discipline to follow in the Open Source world. A truly beneficial, strong and diverse community has to eliminate that thought entirely. The goal is ...

Read More

Simpycity 2.0.0 released (An ORM in Python)

What Simpycity Is

Simpycity is an object-relational mapper. It seamlessly maps PostgreSQL query
and function result sets to Python classes and class attributes.

It allows for the easy and rapid development of query- and
stored procedure-based data representations. Simpycity leverages PostgreSQL's
powerful composite type system, and the advanced type handling of the psycopg2
database access library.

What Simpycity is Not

Simpycity is not a SQL generator and does not attempt to abstract or hide SQL.
Simpycity is designed for developers who deeply understand SQL and
desire to write the best possible SQL representations for their database.
Simpycity also rejects the Active Record paradigm, whose simplistic patterns
fail in even moderately complex systems.

Core Philosophy

The core philosophy behind Simpycity ...

Read More