Postgresql on enumerator.dev

Ouch! Killing PG Performance with Row-Level Security

Thu, 18 Apr 2024 00:00:00 +0000

Postgres 11 and higher has a feature called Row Security Policies, or Row-Level Security, which is a great feature for restricting access to rows in a tabel on a per-user basis.

However, if you’re not careful, you can drag down performance with a bad policy.

The post below comes from a recent project. Now that I’ve researched for this post, I want to benchmark these policies against each other and compare their explain plans. Unfortunately, I am writing on a laptop that doesn’t have much installed, and the environment we were working in when we encountered this issue didn’t have adequate explain plans.

The post below has examples, but no benchmarks. Stay tuned for a followup post!

What is RLS in Postgres?

Let’s start with a policy on a single table. This table is called pets and we’re going to let the current_user who is logged in to the database only view pets that belong to them:

CREATE TABLE pets (
    id SERIAL PRIMARY KEY,
    name TEXT NOT NULL,
    human TEXT NOT NULL
);

ALTER TABLE pets ENABLE ROW LEVEL SECURITY;

CREATE POLICY read_pets ON pets
    FOR SELECT
    USING (human = current_user);

When I run SELECT * FROM PETS I will only get records back that map onto my user. If I try SELECT a pet that does not belong to me, I will get nothing back—as if the record didn’t exist at all.

Performance Foot-Gun

Okay, now let’s shoot ourselvs in the foot with Row-Level Security. Fun!

Let’s create a table called humans that can have many pets through a join table called humans_pets. We’ll also enable Row-Level Security on pets and add in index to the join table:

CREATE TABLE humans (
    id SERIAL PRIMARY KEY,
    name TEXT NOT NULL
);

CREATE TABLE pets (
    id SERIAL PRIMARY KEY,
    name TEXT NOT NULL,
);

CREATE TABLE humans_pets (
    human_id INTEGER REFERENCES humans(id),
    pet_id INTEGER REFERENCES pets(id),
    PRIMARY KEY (human_id, pet_id)
)

ALTER TABLE pets ENABLE ROW LEVEL SECURITY;

CREATE index ON humans_pets(human_id, pet_id);

Now we’ll add a policy that allows humans to only read pets that belong to them. For this example, we’ll assume that current_user_id() is a function that returns the ID of the user we are selecting for.

In order to write this policy, we need to look at the humans_pets table. And here is where we shoot ourselves in the foot!

CREATE POLICY read_pets ON pets
    FOR SELECT
    USING (EXISTS (
        SELECT 1 FROM human_pets
        WHERE human_pets.pet_id = pets.id AND
        human_pets.human_id = current_user_id()
    ));

This policy looks correct because it is ensuring that there is a record that associates the human to the pet. This is all fine and dandy when we run SELECT * FROM pets WHERE id = 1 because we’re only selecting one pet.

But what happens when we run SELECT * FROM pets;? Postgres will compare the policy statement to every row in the table. That means the the USING statement will be executed once per row.

Oops! This is an n + 1! We now query + 1 time for n rows. Not good!

How to Write a Performant RLS Policy

Okay, lets fix this. We can rely a bit on the query planner here and write a policy that uses IN to see if the row exists in the table:

CREATE POLICY pets_self ON pets
    FOR SELECT
    USING (
        id IN (
            SELECT pet_id FROM human_pets
            WHERE human_id = current_user_id()
        )
    );

This approach is much better. It will still call the policy once per row, but at least the query planner will have cached the plan for the SELECT. Here we’re comparing the id to an array, rather than having to select each human_pet record individually.

Can We Make it Faster?

Maybe! The following code is untested, but I want to give it a try. This blog post indicates that using an array of pet_ids is fast. That’s great, we’re already doing that. There’s another trick, though. Under the hood Postgres will treat functions as prepared statements. Which should improve performance when we need to call the function again.

Here’s an example of what that might look like. First we create a function that returns pet IDs.

CREATE OR REPLACE FUNCTION humans_pets_ids(human_id)
AS $$
  SELECT pet_id
  FROM humans_pets
  WHERE (
    humans_id = humans_pets.human_id
  )
$$
LANGUAGE sql;

Then we use that function in our policy.

CREATE POLICY pets_self ON pets
    FOR SELECT
    USING (
        id IN(
            SELECT pet_id FROM humans_pets_ids(
                current_user_id()
            )
        )
    );

In this policy, we’re using the function humans_pets_ids, which might give us a small performance boost thanks to PL/pgSQL.

Conclusion

I’m hoping to have time this week to benchmark gese policies and get some explain plans. I’m really curious to know if the function approach is any faster and I’d like to see explain plans for the first two policies. I’ll write a followup post soon!

DROP TABLE oh_no CASCADE;

Mon, 12 Feb 2024 00:00:00 +0000

A clean database is a good database, right? Right!

We’ll if that’s the case, you certainly want to CASCADE your DROP TABLE so you clean up orphaned objects. But then….oh no! You’ve deleted everything! How does this happen?

Lets start with a very simple users table.

CREATE TABLE users (
    id SERIAL PRIMARY KEY,
    username VARCHAR(255) NOT NULL UNIQUE,
    created_at TIMESTAMP WITH TIME ZONE DEFAULT CURRENT_TIMESTAMP,
    updated_at TIMESTAMP WITH TIME ZONE DEFAULT CURRENT_TIMESTAMP
);

Now these users can have posts (like a blog post):

CREATE TABLE posts (
    id SERIAL PRIMARY KEY,
    title VARCHAR(255) NOT NULL,
    content TEXT NOT NULL,
    CONSTRAINT fk_posts_user_id
    FOREIGN KEY (user_id)
    REFERENCES users(id)
);

Great, looking good! Users have blog posts and our DB knows that user_id belongs to, and must exist in the users table. The last three lines are what makes this magic work. Its saying, verify that each post has a valid owner.

REFERENCES helps us avoid errors where we’re tying to load a post but the user doesn’t exist.

Now, let’s accidentally delete everything!

DROP TABLE users CASCADE;

💥 everything is gone. WHY?

CASCADE tells Postgres to delete all related objects. Since posts has REFERENCES users(id), Postgres knows that posts can’t exist without users, so the DROP TABLE cascades down to posts.

If you want to drop users and preserve posts, you need to remove the constraint on the user_id column before dropping the table:

ALTER TABLE posts DROP CONSTRAINT posts_user_id_fkey;

And then you can drop the table safely like you did before.

DROP TABLE Best Practice

Never run DROP TABLE with CASCADE without first running it without the CASCADE option. Running it without the CASCADE option will tell you which foreign key constraints you need to remove in order to avoid data loss.

One WEIRD Trick for Speeding Up ORDER BY That You Probably Shouldn't Use

Thu, 29 Apr 2021 00:00:00 +0000

I originally posted this on dev.to, and I’m reposting it here because I often link people to it when they are trying to understand Postgres performance issues.

I was digging into a performance issue at work recently and found THE WEIRDEST way of speeding up a query that uses ORDER BY. When testing against our replica database, I was getting results up to 100 times faster than before.

“Too good to be true!” I thought. Well, it turns out that it is sort of too good to be true.

The WEIRD Trick

Here’s an example query that was having issues. We’re selecting some records and ordering by ID.

SELECT *
FROM activities
WHERE account_id IN (999102,989987,809354,100964)
ORDER BY id DESC
LIMIT 10;

In the explain plan, we’re spending almost 9 seconds scanning the index for activities.id 😯

This was baffling to me! We’re using an index, why is this slow?!

As any good developer does, I went digging in Stack Overflow for answers. Unfortunately, I can’t find the original answer I was looking at, but someone had suggested adding + 0 to the ORDER BY clause.

WHAT?! So I tried it.

EXPLAIN ANALYSE SELECT *
FROM activities
WHERE account_id IN (999102,989987,809354,100964)
ORDER BY id + 0 DESC
LIMIT 10;

Limit (cost=32383.82..32383.84 rows=8 width=149) (actual time=71.250..71.252 rows=8 loops=1)
  -> Sort (cost=32383.82..32410.22 rows=10557 width=149 (actual time=71.249..71.250 rows=8 loops=1)
    Sort Key: ((id + 0)) DESC
    Sort Method: top-N heapsort Memory: 26kB
    -> Index Scan using index_activities_on_account_user_id on activities (cost=0.57..32172.68 rows=10557 width=149) (actual time=2.292..71.154 rows=132 loops=1)
      Index Cond: (account_user_id = ANY ('{999102,989987,809354,100964}'::integer[]))
Planning Time: 0.122 ms
Execution Time: 71.279 ms

🤯 71.279 ms!!!!!

What in the ACTUAL is happening here?

Well, first, you should know that this table is pretty big. There are just under 100,000,000 records in it. AND, you should know that in some cases, this query can return LOTS of records without the LIMIT in there. In the EXPLAIN plan screen shot above, you can see that the query found just under 9,000 records, but couldn’t apply the LIMIT until they were sorted.

Another thing to know here is that this table is busy. We add almost 1,000,000 records in a day. Which means that those 9,000 records have IDs that are pretty far apart. There are big gaps between the numbers returned in the ID column.

Imagine this query returned 100 records, but the IDs were more than 100 apart. E.g., IDS 100, 200, 300…etc. When postgres is trying to sort these records it does a reverse scan through the index and has to read the order of EVERY number including the ones not returned by the query.

So what is this magic + 0 doing? Adding + 0 to the ORDER BY clause forces Postgres to load the records into memory and sort them without the index, which is way faster. You can see in the explain plan above, that the sort in memory was super fast.

Why You Probably Shouldn’t Do This in Production

This really depends on your data. I knew that only a small percentage of users were experiencing really slow requests due to this query, but I didn’t know what the impact of the + 0 magic would be on users that have fewer activities records. So I ran an experiment using GitHub’s Scientist gem.

The results of the experiment are still coming in, but so far, the original ORDER BY is almost tied with ORDER BY id + 0 for average execution time. However, the 95 percentile execution time (meaning 5% of queries are slower than this number), are way out of sync! ORDER BY id + 0 is almost twice as slow in the 95 percentile category.

The results of the experiment tell me that overall, ORDER BY with an index scan might cause timeouts for some users, but ORDER BY id + 0 with no index scan seems to cause slower queries overall for most users.

Looks like I’m going to have to find a different solution to this problem.

Conclusion

Adding + 0 to your ORDER BY id is a wacky trick that will speed up queries when the index is HUGE and the IDs returned have big intervals.

However, it won’t always work. Indexes are still fast! Test your queries safely in production to make sure that you’re not making a worse experience overall to save a few slow queries.