Question

I am a ruby-junior. My app allows a user to enter Contacts and/or upload a CSV file.

Running on my local branch, if I add a Contact - the Contact gets added from View, Controller & dBase and there it works great.
If I then allow the user to Import CSV file. It starts to import the file. However, the User is now unable to Add a Contact via the App. It essentially hangs until the CSV import is completed.

I am using the following versions:

  ruby "2.3.0"
  gem "rails", "4.2.5.1" gem "pg", "0.17.1" # postgresql database 
  gem "delayed_job_active_record", ">= 4.0.0.beta1" # background job
  processing gem "delayed_job_web", ">= 1.2.0" # web interface for delayed job

Also using:

> class CsvUploader < CarrierWave::Uploader::Base

def store_dir "uploads/#{model.class.to_s.underscore}/#{mounted_as}/#{model.id}" end

Here is the worker:

class ImportCsvFileWorker

def self.perform(csv_file_id) csv_file = CsvFile.find(csv_file_id)
csv_file.import!
csv_file.send_report!   end
end

I am using smarecsv parsing service

def process_csv parser = ::ImportData::SmartCsvParser.new(csv_file.file_url)

parser.each do |smart_row|
  csv_file.increment!(:total_parsed_records)
  begin
    self.contact = process_row(smart_row)
  rescue => e
    row_parse_error(smart_row, e)
  end
end   rescue => e # parser error or unexpected error
csv_file.save_import_error(e)   end

Does delayed_job lock the dbase for the User/Contact so i can't add any Contacts via the App?

Locally, the app is frozen/hanging or seems locked until background delayed_job is completed (BTW if i run on Heroku, it causes H12 errors but figure I need to fix the issue locally first). Just trying to understand - what is causing it to be locked? Should it be doing this? Is it code (the business logic of the CSV file and the View of Adding a Contact both work independent)? But the App side will not work if there is a background job running or is it the way Active Record handles it. Is there a way around this?

I have not isolated it but am suspicious that if any background job is running, the app becomes unavailable.

I have tried to include all the relevant facts - let me know if any further details needed. many thanks for help.

UPDATE - i discovered i have a ContactMergingService that seems to locking all the contacts. If i comment out this service below, my application does not hang.

So my question is what are other options = Before adding a Contact, what I am trying to do is find all existing same email address (if I find it, I append contact details). how do i do this without locking dbase?

is it because I am using 'find' method? is there a better way?

> class ContactMergingService
> 
>   attr_reader :new_contact, :user
> 
>   def initialize(user, new_contact, _existing_records)
>     @user = user
>     @new_contact = new_contact
>     @found_records = matching_emails_and_phone_numbers   
>   end
> 
>   def perform
>     Rails.logger.info "[CSV.merging] Checking if new contact matches existing contact..."
>     if (existing_contact = existing_contact())
>       Rails.logger.info "[CSV.merging] Contact match found."
>       merge(existing_contact, new_contact)
>       existing_contact
>     else
>       Rails.logger.info "[CSV.merging] No contact match found."
>       new_contact
>     end   end
> 
>   private
> 
>   def existing_contact
>     Rails.logger.info "[CSV.merging] Found records: #{@found_records.inspect}"
>     if @found_records.present?
>       @user.contacts.find @found_records.first.owner_id # Fetch first owner
>     end   end
> 
>   def merge(existing_contact, new_contact)
>     Rails.logger.info "[CSV.merging] Merging with existing contact (ID: #{existing_contact.id})..."
>     merge_records(existing_contact, new_contact)   end
> 
>   def merge_records(existing_relation, new_relation)
>     existing_relation.attributes do |field, value|
>       if value.blank? && new_relation[field].present?
>         existing_relation[field] = new_relation[field]
>       end
>     end
>     new_relation.email_addresses.each do |email_address|
>       Rails.logger.info "[CSV.merging.emails] Email: #{email_address.inspect}"
>       if existing_relation.email_addresses.find_by(email: email_address.email)
>         Rails.logger.info "[CSV.merging.emails] Email address exists."
>       else
>         Rails.logger.info "[CSV.merging.emails] Email does not already exist. Saving..."
>         email_address.owner = existing_relation
>         email_address.save!
>       end
>     end
>     new_relation.phone_numbers.each do |phone_number|
>       Rails.logger.info "[CSV.merging.phone] Phone Number: #{phone_number.inspect}"
>       if existing_relation.phone_numbers.find_by(number: phone_number.number)
>         Rails.logger.info "[CSV.merging.phone] Phone number exists."
>       else
>         Rails.logger.info "[CSV.merging.phone] Phone Number does not already exist. Saving..."
>         phone_number.owner = existing_relation
>         phone_number.save!
>       end
>     end   end
> 
>   def matching_emails_and_phone_numbers
>     records = []
>     if @user
>       records << matching_emails
>       records << matching_phone_numbers
>       Rails.logger.info "[CSV.merging] merged records: #{records.inspect}"
>       records.flatten!
>       Rails.logger.info "[CSV.merging] flattened records: #{records.inspect}"
>       records.compact!
>       Rails.logger.info "[CSV.merging] compacted records: #{records.inspect}"
>     end
>     records   end
> 
>   def matching_emails
>     existing_emails = []
>     new_contact_emails = @new_contact.email_addresses
>     Rails.logger.info "[CSV.merging] new_contact_emails: #{new_contact_emails.inspect}"
>     new_contact_emails.each do |email|
>       Rails.logger.info "[CSV.merging] Checking for a match on email: #{email.inspect}..."
>       if existing_email = @user.contact_email_addresses.find_by(email: email.email, primary: email.primary)
>         Rails.logger.info "[CSV.merging] Found a matching email"
>         existing_emails << existing_email
>       else
>         Rails.logger.info "[CSV.merging] No match found"
>         false
>       end
>     end
>     existing_emails   end
> 
>   def matching_phone_numbers
>     existing_phone_numbers = []
>     @new_contact.phone_numbers.each do |phone_number|
>       Rails.logger.info "[CSV.merging] Checking for a match on phone_number: #{phone_number.inspect}..."
>       if existing_phone_number = @user.contact_phone_numbers.find_by(number: phone_number.number)
>         Rails.logger.info "[CSV.merging] Found a matching phone number"
>         existing_phone_numbers << existing_phone_number
>       else
>         Rails.logger.info "[CSV.merging] No match found"
>         false
>       end
>     end
>     existing_phone_numbers   end
> 
>   def clean_phone_number(number)
>     number.gsub(/[\s\-\(\)]+/, "")   end
> 
> end

Answer 1

You can try something like:

 Thread.new do
    ActiveRecord::Base.transaction do   
      User.import(user_data)
    end
    ActiveRecord::Base.connection.close
 end

In your CVS importing code.

Answer 2

我们得出结论，问题的原因是当CsvParsingService＃perform运行时，它将AccessShareLocks放在数据库中的某些表上（我们认为是Contacts，EmailAddresses，PhoneNumbers，也许是用户）。

锁定一直持续到方法完成。尝试访问其中一个锁定表的任何其他请求只会等待数据库解锁。由于该方法会解析给定上载的csv_file的每一行，因此运行时间长达90分钟。

尝试访问其中一个锁定表的应用程序的任何请求都将停止并等待表解锁。因为Heroku会在30秒后切断请求，这就是产生H12错误的原因（在应用程序端）。

问题的原因是gem state-machine_active记录默认包装事务中的每个状态转换。

工作者通过运行csv_file.import！来调用解析服务，这会触发csv_file状态机的转换，然后调用CsvParsingService并解析每一行。由于状态机将所有内容都包装在事务中，因此在状态转换完成之前不会执行任何操作。

通过将gem更新为版本0.4.0pre，并将选项use_transactions：false添加到CsvFile模型中的状态机，它在调用.import时不再锁定数据库！和处理。

Worker hijacking Active Record model

2 个答案: