Blog

Scale 14x, PostgreSQL mini-conf, PgConf.US and NYCPUG

It is really not fair to call it a mini-conf. The Scale 14x, PostgreSQL Day attendance was larger than every conference except PgConf.US (EDIT: in the United States/Canada). It is a great opportunity to integrate with a wider community that is diverse, technologically capable and at the front lines of production installations.

I spoke on Backups: The Good, the Bad, and the Ugly. I had over 100 attendees in my room. It was obvious throughout the talks that this is going to be the new West coast conference for PostgreSQL. The opportunity for advocacy and integration into alternative technologies is just too great to ignore from either PostgreSQL or the wider FOSS community that attends SCALE.

This is ...

Read More

13:58, the sun is shining, the sky is blue, and I just had the best tomatoes -- ever. Welcome to Vienna and PgConf.eu

I am sitting in a glorious (although not the conference) hotel, writing this article. It is 13:58, at least it is where I am from, the great Pacific Northwest. I must admit, I miss the trees although I hear there are lots of them if you leave the city. What is there to say about this great city that the European community picked for their latest conference?

First and probably obvious, it is full of history. Second, don't bother doing anything on a Sunday. Third, smoking is allowed in restaurants (one ridiculous thing I have noted). Fourth, they sell water that has, "Natural Oxygen". Fifth, it feels a lot like Paris but the people seem friendlier. Lastly, this ...

Read More

PostgreSQL 9.5, Community, Features and More!

I spoke at the Whatcom PUG meeting last night on PostgreSQL 9.5. This is my fourth time giving this talk. The previous locations were DCPUG, PhillyPUG, and NYCPUG last month. The talk was well received and this was Whatcom PUG's best turn out yet! We even had an Open Street Map developer visit from Vancouver B.C.

The presentation does discuss some of the more popular features of 9.5, but as a whole it discusses the state of PostgreSQL as of 9.5. That includes features, community, and process. I think the most important item is the user interaction. At each presentation location I brought up the fact that PostgreSQL has no bug/issue tracker. This led ...

Read More

Tip for West side U.S. folks going to PgConf.EU in October

This tip works very well for me because of my physical location (Bellingham, WA) but it would also work reasonably well for anyone flying from Denver->West Coast including places such as Houston. It does take a little bit of patience though.

A normal trip for myself would mean driving down to SEA which is 90 minutes to 2 hours. This year, I decided on whim to see what it would take to fly out of YVR (Vancouver, B.C.) which is only 60 minutes driving.

Since I would be flying out of YVR on a non-connecting flight, I paid Canadian Dollars. For those that haven't been paying attention, the U.S. dollar has been doing very well lately ...

Read More

Elevating your confidence with the Elephant's restoration capabilities

In the beginning

There was Unix, Linux and Windows. They all run on hardware and that hardware all has bugs. What is the best way to work around hardware bugs? Backups.

You haven't had bad hardware, only bad developers? That's o.k., we have a solution to them too. It is called backups.

You haven't had bad hardware or bad developers, just bosses who still demand to have direct access to the data even though they haven't proven an ability to extract useful information without an extreme amount of hand holding? That's o.k., we have a solution to them too. It is called backups.

You haven't had any of the above? Lucky you ...

Read More

A new user discovers the PostgreSQL public schema

A new user of PostgreSQL recently discovered that PostgreSQL allows any PostgreSQL user to create objects and data by default.[1] I know you are saying, "What... PostgreSQL has some of the most advanced and flexible security in the industry!" and you are absolutely correct, we do. However, once you can connect to PostgreSQL, you have some interesting default capabilities. Consider the following example:

postgres@sqitch:/# psql -U postgres
psql (9.2.11)
Type "help" for help.

postgres=# create user foo;
CREATE ROLE
postgres=# \q

No biggy, we created a user foo as the super user postgres. All is good. However, what can that user foo do?

postgres@sqitch:/# psql -U foo postgres
psql (9.2.11)
Type "help" for ...

Read More

Let's delete contrib!

There has been a lot of discussion about the upcoming extension pg_audit and whether or not it should be in contrib. You can read about that here. The end result of the discussion is that pg_audit is going to be reverted and not in contrib. There were plenty of technical reasons why people didn't want it in contrib but I have a different reason. It is an extension. It doesn't need to be in contrib. In fact, I argue that because of pgxs and extensions we don't need contrib at all. If you don't follow the mailing lists my argument is blow and please feel free to comment here. The discourse is very much needed on ...

Read More

Updating the .Org docs on backups

I spent a great deal of time working through the SQL DUMP portion of the 9.5devel docs this past week. Below is the current text of what I have and it would be great if my readers would take a look and offer some thoughtful feedback. What would you like to see added? What would you like to see changed? Please note that this is reference documentation not tutorial documentation.

This is just the straight HTML dump that is generated from Docbook but since it is inline the links won't work. The current -devel docs are here and the updated version I am working is below:


24.1. SQL Dump

PostgreSQL provides the program pg_dump for generating a ...

Read More

WhatcomPUG meeting on 04/21. Start date, end date, calculate

The PUG meeting was good. We now have a consistent if small group that are attending. Before the presentation we spoke about possibly moving the group to meetup to get a little better visibility. G+ Communities are awesome but Meetup seems to be where the people in the area look.

The presentation was provided by Eric Worden who happens to be a CMD employee. The talk overall is very good and provided a lot of information that I didn't know about dates. It also lead to the development of a new PostgreSQL extension (more on that at a later time).

The most interesting part of the talk to me was the use of a dimensions table to make date ...

Read More

Reflections on PgConf.US 2015

Saturday the 18th of April, I woke up to the following:
 

It was one of those moments that you realize just how blessed of a life you have. A moment where you stop and realize that you must have done something right, at least once. I was with my all of my ladies, there were no other people at the camp site, the weather was clear and it was set to hit 68F. The only sound was the gentle lapping of water and the occasional goose.

It was at this time that I was able to finally take a step back and reflect on PgConf.US. This conference meant a lot to me professionally. I didn't organize it ...

Read More

WhatcomPUG meeting last night on: sqitch and... bitcoin friends were made!

Last night I attended the second WhatcomPUG. This meeting was about Sqitch, a interesting database revision control mechanism. The system is written in Perl and was developed by David Wheeler of PgTap fame. It looks and feels like git. As it is written in Perl it definitely has too many options. That said, what we were shown works, works well and appears to be a solid and thorough system for the job.

I also met a couple of people from CoinBeyond. They are a point-of-sale software vendor that specializes in letting "regular" people (read: not I or likely the people reading this blog) use Bitcoin!

That's right folks, the hottest young currency in the market today is using the ...

Read More

Stomping to PgConf.US: Webscale is Dead; PostgreSQL is King! A challenge, do you accept?

I submitted to PgConf.US. I submitted talks from my general pool. All of them have been recently updated. They are also all solid talks that have been well received in the past. I thought I would end up giving my, "Practical PostgreSQL Performance: AWS Edition" talk. It is a good talk, is relevant to today and the community knows of my elevated opinion of using AWS with PostgreSQL (there are many times it works just great, until it doesn't and then you may be stuck).

I also submitted a talk entitled: "Suck it! Webscale is Dead; PostgreSQL is King!". This talk was submitted as a joke. I never expected it to be accepted, it hadn't been written ...

Read More

PostgreSQL is King! Last week was quite busy being a servant.

Last week was one of the busiest community weeks I have had in a long time. It started with an excellent time in Vancouver, B.C. giving my presentation, "An evening with PostgreSQL!" at VanLUG. These are a great group of people. They took all my jibes with good humor (Canadians gave us Maple Syrup, we gave them Fox News) and we enjoyed not only technical discussion but discussions on technology in general. It is still amazing to me how many people don't realize that Linux 3.2 - 3.8 is a dead end for random IO performance.

After VanLUG I spent the next morning at the Vancouver Aquarium with my ladies. Nothing like beautiful weather, dolphins and jelly ...

Read More

AWS performance: Results included

I am not a big fan of AWS. It is a closed platform. It is designed to be the Apple of the Cloud to the Eve of Postgres users. That said, customers drive business and some of our customers use AWS, even if begrudgingly. Because of these factors we are getting very good at getting PostgreSQL to perform on AWS/EBS, albeit with some disclosures:
  1. That high IO latency is an acceptable business requirement.
  2. That you are willing to spend a lot of money to get performance you can get for less money using bare metal: rented or not. Note: This is a cloud issue not an AWS issue.

Using the following base configuration (see adjustments for each configuration after ...

Read More

Don't kill yourself

As a PostgreSQL consultant you end up working with a lot of different types of clients and these clients tend to all have different requirements. One client may need high-availability, while another needs a DBA, while yet another is in desperate need of being hit with a clue stick and while it is true that there can be difficult clients, there is no bad client.

What!!! Surely you can't be serious?

Don't call me shirley.

I am absolutely serious.

A bad client is only a reflection of a consultants inability to manage that client. It is true that there are difficult clients. They set unrealistic expectations, try to low ball you by with things like: "We can get ...

Read More

Along the lines of GCE, here are some prices

I was doing some research for a customer who wanted to know where the real value to performance is. Here are some pricing structures between GCE, AWS and Softlayer. For comparison Softlayer is bare metal versus virtual.

GCE: 670.00
16 CPUS
60G Memory
2500GB HD space

GCE: 763.08
16 CPUS
104G Memory
2500GB HD space

Amazon: 911.88
16 CPUS
30G Memory
3000GB HD Space

Amazon: 1534.00
r3.4xlarge
16 CPUS
122.0 Memory
SSD 1 x 320
3000GB HD Space

Amazon: 1679.00
c3.8xlarge
32 CPUS
60.0 Memory
SSD 2 x 320
3000GB HD Space

None of the above include egress bandwidth charges. Ingress is free.

Softlayer: ~815 (with 72GB memory ~ 950)
16 Cores ...

Read More

GCE, A little advertised cloud service that is perfect for PostgreSQL

Maybe...

I have yet to run PostgreSQL on GCE in production. I am still testing it but I have learned the following:

  1. A standard provision disk for GCE will give you ~ 80MB/s random write.
  2. A standard SSD provisioned disk for GCE will give you ~ 240MB/s.

Either disk can be provisioned as a raw device allowing you to use Linux Software Raid to build a RAID 10 which even further increases speed and reliability. Think about that, 4 SSD provisioned disks in a RAID 10...

The downside I see outside of the general arguments against cloud services (shared tenancy, all your data in a big brother, lack of control over your resources, general distaste for $vendor, or whatever else ...

Read More

PDXPGDay 2014

I had the honor of being asked to give the introduction at PDXPGDay 2014 this past Saturday. I didn't speak very long but it was great to see a lot of the old stomping ground. It had been quite some time since I had been in the group of Wheeler, Roth, Wong, Berkus and a few others.

The conference was really a mini-conference but it was great. It was held in the exact same room that PostgreSQL Conference West was held all the way back in 2007. It is hard to believe that was so long ago. I will say it was absolutely awesome that PDX still has the exact same vibe and presentation! (Read: I got to wear ...

Read More

apt.postgresql.org... a wonderful if flawed apt repository

The site apt.postgresql.org is a great resource for those who live in the Debian derived world. It keeps up to date with the latest postgresql packages and has a whole team dedicated to creating these packages. Of course, this is the Open Source world so not everyone agrees 100% with the way things are done in this project. As I noted here, there are some issues.

These issues are not to detract from otherwise excellent work but a note to those who use the repository to look for further problems. I also have a video displaying specifically what the issues are, here.

Read More

Kicking the Donkey of PostgreSQL Replication

This is the title of a talk I am developing for the matured PostgreSQL Conference: PGConf NYC 2014 . Formerly a PgDay, this is now a full blown conference extending two days with three tracks. From all reports it is set to be the largest PostgreSQL Conference ever in the United States, surpassing even the old West and East series (which no conference in the U.S. has done to date). It is truly exciting times for our community.

This talk will be a departure from my standby talks of PostgreSQL Performance and Choosing the right hardware. Katz asked me, "to bring your full East Coast from the West Coast personality.". I plan on doing so. So cinch up the boot ...

Read More

PgConf.eu is over, it was a blast but I am curious about the future

First let me say that I attended pgConf.eu like I attend every conference (that I am not running). I show up for a few hours on the first day, then I come back and attend my talk. I don't take travel lightly and as much as I bromance my fellow elephant bretheren, I want to explore the sights and this was freaking Ireland people.

I had an odd feeling for the time I was there. The community was in full force, there was at least 240 people there and that was great. It was the commerce side, the sponsor side, the **money** side that was lacking. EnterpriseDB, Cybertec and 2ndQuadrant were there with booths but I wonder if ...

Read More

Just back from NYCPug August, on to more talks

In August I spoke at NYCPUG on Dumb Simple PostgreSQL Performance. The talk was well accepted and there was about 60 people in attendance. I have always enjoyed my trips to NYC but this is the first time I have taken a leisurely look at the city. I found myself enjoying a water front walk from 42nd, through the Highline, to Battery Park, all the way to the Brooklyn Bridge and over to Brooklyn to a great pub for dinner. What I enjoyed most about the walk outside of the 10 miles was the community that was present. I think it is easy to get jaded by "midtown" and all that is the tourist in that area. The hustle and ...

Read More

Compiling and installing OpenSRF 2.2 on Centos 5.9

We do quite a bit of work for King County Library systems. The library system has 45 branches and runs the Open Source Evergreen ILS. One of the very smart things that the Evergreen project decided was that their database of choice would be PostgreSQL. One of the things that the Evergreen project is not good at is supporting LTS releases of Linux and therefore certain things can be a chore. For example, by default OpenSRF 2.2 which is the current stable OpenSRF release can not be installed via RPM or compiled from source by default on CentOS 5.9.

When discussing with the community about CentOS, the response was the classic responses of, "just upgrade", "move to Fedora ...

Read More

Calling Bullsh*t in Open Source communities

We are all human. We all lose our temper. We all have our moments of, "I really wish I could take that back". Of course not if you are not Linus Torvalds. Now everyone knows that Linus has a temper, that he is a foul mouth, lacks certain social graces and is generally one of the, if not the most important developers to surface in the last 20 years. Does that mean he gets to be a jerk? In his mind, yes.

In some ways I agree with Linus. If you are a donkey butt and you don't pay attention to your community and follow its guidelines, then just leave. We don't have time for you anyway. We ...

Read More

postgres_fdw for 9.2

We have backported the postgres_fdw to 9.2. It is read only of course as the infrastructure for writes is not in 9.2 but it is usable. Enjoy it!

  • Postgres-FDW
  • Read More