Postgres - query large data set with order in batches using postgres cursor rails

If the table in postgres has large amount of data, you should avoid loading records all at once. In that case, batch processing methods ActiveRecord#find_each allow you to work with the records in batches, thereby greatly reducing memory consumption.

The disadvantage of using ActiveRecord#find_each is that it's not possible to set the order of records in the query, it automatically sets the order of records to ascending on the primary key (“id ASC”).

If you want the records in some other order than primary key you should be using postgresql_cursor. However, postgresql_cursor does not natively support eager loading. If you need to efficiently preload an association you can trigger it manually:

batch_size = 1000
data = Model.order("field")

data.each_instance(block_size: batch_size).lazy.each_slice(batch_size) do |batch|, [:associated_models])
    batch.each do |source_object|
      your logic goes here

keep Coding !!!


What do we do?

We work on website development using Ruby on Rails, NodeJS, ReactJS, AngularJS. View all the services we provide here.

Contact us See our work