CODEX

In this article, I present a case study of adopting the background job and queue model to a real system. Given the current system architecture, an improved version is proposed and the details of designing such the system from different aspects will be provided. If the whole content did not make much sense, I highly recommend reading my other articles in the following order:

• Conceptual and Practical
• Practical Application Use Cases
• Issues of Concurrency and Execution Order

Input

A notification system written in Python and MySQL has several performance issues, that are

• Latency that slows down services
• High service demand during…

CODEX

This post aims to provide you an overview of how to fulfill requirements from the perspective of system design. Having said that, its content is not technology-driven, meaning that the implementation of what discussed is transparent to something like Kafka or whatever message queues like RabitMQ. There are no such features specific to a certain system or technology or framework.

In my previous articles (here and here), I did not discuss the relationship between jobs, assuming that they are non-related. By doing so, I simplified things, can do any scale up/down/out on the Worker Nodes as many as you want…

In my previous article, I introduce the concepts of queue and background jobs via an illustrative pharmacy example, its application in practice together with advantages and limitations. This article is part two of it, reviewing different job types and how it comes to the picture via several common applications.

Job Definition and Categories

There are a couple of background jobs, depending on how and on which kind of data they perform. I suggest you find the details in [1]. Here I have summarized some important aspects and practical use cases that you can imagine how background jobs look like.

In general, background jobs are…

Just a kind note: Since Medium does not support maths equations, many of them in this post were changed to raw representation. In case you dislike reading them (as I do), you can find better content here.

Introduction

Previously, we introduced collaborative filtering (CF) recommendation system based on the behaviors of users or items in the neighborhood. At this time, another approach to build a CF system is presented — via solving the matrix factorization (MF) or matrix decomposition problem. It is called matrix factorization collaborative filtering (MFCF).

Recall that for content-based recommendation systems, each item is represented by a vector…

Just a kind note: Since Medium does not support maths formula, many of them in this post were changed to raw representation. In case you dislike reading them (as I), you can find better content here.

Introduction

I introduced Content-based Recommendation Systems in the last article, which aims to provide item recommendations to the users based on the item’s features. Basically, the content-based approach builds a model for every user interest regardless of the presence of other users but items’ profiles. The advantage of this approach lies in its requirement of memory resources and computation time. It leverages the features of…

Introduction

As a netizen, probably you have met the following facts many times:

• Youtube automatically plays clips related to the one that you are watching. And especially when you are at its homepage, Youtube also automatically suggests clips that you find interesting.
• Have you ever noticed the section [Frequently bought together] while doing shopping on Amazon?
• Have you ever wondered that Facebook is secretly listening to your conversation and display the ads relevant to your favorite items even though they are only in your mind? Or friend suggestions?
• Netflix suggests movies to its users.

There are actually many examples of systems…

CODEX

In this article, I introduce basic concepts of a Recommendation System (RS) and 2 basic approaches of building an RS, namely Content-based and Collaborating Filtering. This time we dig into the implementation of a simple RS using Collaborating Filter approach using the item-based technique. You can The user-based one is

Collaborating Filtering (CF)

As mentioned in my previous article, CF is built based on the assumption of shared interests among different individuals. You may think about the law of large numbers and theories of crowd behavior. In a nutshell, it is stated as follows:

If 2 users u1 and u2 review n similar…

At a pharmacy store

Here is what happens at the pharmacy store when an order comes:

• The pharmacist receives your prescription and goes back to the store to get the prescribed medicine
• Bring back to you and tell you how much
• Payment and change (if any and if cash)

Then another guy comes, the pharmacist repeats that process. And another 3 guys come, he still does that one by one. But when 5–10 guys come which might happen a couple of times during a day, he is definitely screwed up.

And if you told him something like “dude, why don’t you just get 1…