Skip to main content

Posts

Showing posts from November, 2024

Using Ruby on Rails and AWS SQS for Website Crawling

Recently, I returned to Ruby on Rails , one of my favourite web application frameworks. This time, I aimed to build a simple management tool for scheduling website crawls—a key component of a side project I'm working on. The tool's purpose is straightforward: Maintain a list of websites to crawl. Schedule crawls for these websites. Trigger a Golang process for the actual crawling task. To facilitate communication between the Rails app and the Golang service, I chose AWS SQS (Simple Queue Service). SQS provides a reliable way to send, receive, and manage messages between distributed systems. Adding an SQS Service in Rails In Rails, services are often used to encapsulate business logic that doesn’t belong in the standard MVC structure. For my application, I created a services/sqs_send_service.rb to handle sending messages to SQS queues. Here’s the implementation:   require "aws-sdk-sqs" class SqsSendService # Client is a class method that ...