Upon designing the Basar platform

The last blog entry about designing an asynchronous ecommerce platform dates back to March 2013 which is ... quite a while. As things slightly changed a bit since than (I switched departments, projects where abandonded and so on) I was unfortunately too busy to work on that topic intensively. With some fresh new ideas, encouragement gained from personal talks and some interesting hints on technology retrieved from talks at this year JavaZone, I will get into more detailed work during the upcoming weeks.

Actually, there are two things, I would like to talk about in this blog input: (1) project name change and (2) design decisions.

The first topic is short and has a short explanation as well: I changed the projects name from iServices to Basar as it - from my perspective - expresses the intention of the platform much better than the former name. But names are just sound and smoke, therefore read on for some architectural facts whereas you can find the conceptual aspects in another blog post.

For several reasons I always found the idea of having loosely coupled services formed into a larger integrated system very intriguing. Amongst others these are my arguments:

1. clear separation of concerns, even across teams
2. only a small number of global design requirements along with a larger responsibility mapped to the teams
3. perfect match of technologies applied inside components to solve local problems - no need to swing the hammer
4. easy exchangeability of features
5. easy extensibility

For some more - very good - reasons see the video of James Lewis about "Micro-Services - Java, the UNIX way held at JavaZone 2013.

But where does loosely coupling start and where does it come to an end? Is it just like throwing some guys into the ring which have never met before or is there a common foundation they all adhere to like common frameworks or concepts? .... as always: it depends on the context. Or to say it with the words of Harold Samuel: location, location, location

In this case, the desired goal to provide a mixture of SaaS and PaaS (as a combined product) along with a heavy need to decouple the single building blocks, I decided to go for the communication as common foundation. Rather than on a technical level I choose the conceptual/business level to define the common base. Therefore all services belong to service types which in turn have a defined communication protocol. If a new component wishes to add some recommendation engine features it must adhere with the requirements defined for reco engines inside the platform ... or ... for any good reason, extend the platform and provide a version 2 of the recommendation engine service type.

What we have so far is a set of textual service representations, the desire to leave components in a certain type of isolation and provide the chance to extend the platform if required. Now comes the technology challenge: what paradigm to follow and which framework to use.

As I have stated in an earlier post the platform will definitely be build on an asynchronous foundation. I believe that asynchronous information processing offers the opportunity to leverage the existing hardware at its best - well, you can still mess things up but achieving the opposite is made easy through the paradigm itself.

Another decision made on the path of technology challenge is to let the asynchronous communication be carried out through message passing along with the strong requirement that particular context knowledge must be represented through the message state only. What does that mean, does the platform not have any clue about which reality it represents? No, that's wrong. The building blocks/components have a state but that state is independent from any special context represented inside a message. The message is used to aggregate knowledge from the component which in turn is incorporated into the context of the message but the component does not hold any specific state for any specific message.

Take a recommendation engine for example. It knows about customers, products and which products to offer to which customer. When a customer message requesting a recommendation comes in, it knows - depending on the message state - which data to respond. The component has its own state but is not dependent on the message nor does it keep anything that is bound with the message. However, the message might initiate a state change but that in turn is global and visible to all upcoming messages.

To draw a picture on how I see the platform imagine a large set of chaotically ordered neurons where each neuron may be connected with an arbitrary number of other neurons. A message put into this mesh is ping-ponged between different neurons, each carrying out the task requested until - if requested - the response finally is spilled out to the calling party. Along its path it may have an impact on data or state provided through different neurons, like changing the set of recommendable products as the message told everyone that the last pair of Levi's just left the stock.

Now over to the final stept: which framework to use?

Over the last two years I became fascinated by the Akka framework and its simplicity towards implementation and application management. As it provides me with all required building blocks I decided to go after it and use it as foundation ground.

Now that I have defined the paradigm to follow, the communication methodology to use and the framework to apply: how does this may fit with the idea to have a loosely coupled system as the messages needs to be passed around ... somehow ... and the natural answer would be to use an Enterprise Service Bus? The latter would couple things again more tightly, may become a bottleneck AND is so old skool, thus I need to find another approach.

The challenge was to bring together message passing and loosely coupling along with distributed components. Somehow it is like mashing up parts of Amazons two-pizza rule together with message passing and loosely coupled components into a new approach.

The idea is to have components that accept certain messages and produce defined output - both known to the entire platform and defined on service specification level. These components register themselve on a central instance which keeps track on which service type is implemented by which service component. Additionally it knows in which cases a specific component accepts incoming messages of a certain type. If a service wishes to communicate with another service it looks up its actor reference from the central phonebook, checks which messages it accepts (eg. customer age must be 25) and starts communicating.

Having the core foundation layed out the upcoming posts will address different problems - technical as well as conceptual ones - that needs to be solved to have a consistent architecture.

Crowd Commerce - What the heck....

After a conversation with a friend of mine and some people at work, the question intrigued me, on how to extend the sales of product owners on the one hand and those of web shops owners on the other hand (see my last blog entry on that topic).

During the following days and weeks, a lot of concepts and ideas kept floating through my mind, never knowing how to canalize it in a suitable way. Some of them are so far from the edge that even pros in this field throw a squint on me. Therefore I decided to write them down and see what you people think about them.

The core idea behind it, I have already outlined in my previous post, can be summarized best by the term Crowd Commerce. It all dates back to a book I once read: The wisdom of crowds by James Surowiecki. I got intrigued by the idea that the crowd is one of the most powerful organizational forms and that the crowd is more knowingly than its highest rated member. It is nothing really new, but someone wrote it down in a noteworthy book and people became aware of it.

So, I started to ask myself the question, what would happen if product owners would mainly focus on what they are good at: obtaining products at the best price and handle the sales process. And what would happen if web shops or website owners simultaneously would focus on what they are good at: presenting and recommending products to their customers. Obviously, there would be a gap between both as the product owners miss a suitable way of presenting and selling products whereas the shop owner lacks the ability to buy products at best rates from their producer.

That automatically led me to the question on what needs to be done in order to close that gap and what would happen then? I guess that we would see lots of (small) web shops popping up, having a quite specialized product portfolio they (might) enhance with additional services like suggestions on how to use a specific product or provide construction services.

There are indeed some open questions left to answer, like the handling of delivery costs when odering products belonging to different product owners. Although they were ordered from within a single web shop the customer has to pay each product owner for shipping its product. But from my point of view, that is just a minor problem to solve.

A major topic might be to answer the question if product owners and web shop owners are really willing to give up the area the other part is responsible for: buying products and selling products. But in doing so both contribute to the crowd and therefore to crowd commerce as they do what they are best at. Depending on how their cooperation is defined, the overall crowd leverages its possibilities at its best and all participating parties will win.

As described in the previous blog entry, the iServices platform is a technical approach to translate the ideas of crowd commerce into a platform.

Stay tuned for more information....or get in contact with me, if you are interested in participating ;-)

iServices - Enabling Crowd Commerce

My last blog entry discussed the different options on how to define functional processes within the iServices application. As I have been too vague on the context the application will be implemented for, this article gives you a rough overview on the ideas, concepts and features thought of so far.

It all started with a beer and a discussion with a friend of mine about an idea for a web shop. At its core heart it was quite the same where the products would come from as the set of services surrounding the product portfolio were its unique selling point. One approach provided that we would use one of the several affiliate services and assemble the product catalog from it. This was accompanied, however, that we had to stick to the rules of such services, eg. forwarding the customer to the product owning shop for closing the ordering process (see shopping24.de for example). But that was not we were looking for, we wanted to have an independent shop where the ordering process would be transparent for the customer regarding the product owning web shop or the deliverying party.

The following days the idea intrigued me further since it seem to be an interesting approach for product owners and web shops (or alike) equally. Product owners, even small ones, could provide their catalogs along with any additional information to an arbitrarily large number of web shops/sellers, whereas web shops/sellers can select products, even those they would probably have no access to as their purchase quantity would be far to small, from a large unified catalog. For both it would be somehow a win/win situation.

In my mind, these thoughts formed the picture of a busy bazaar or hypermarket where participants trade with each other equally. Due to the concentration of supply and demand, all participating parties benefit from this setup. While thinking on how to summarize this picture at its best, the term Crowd Commerce came to my mind.

As I still was searching for a context that I could use for showing that asynchronous architectures are suitable for ecommerce systems as well, it was quite obvious that the scenario described about would be the perfect match. It was then that the idea of something like a bazaar or marketplace connecting product suppliers and sellers in n-to-m relationships for increasing sales numbers for both participants was born.

The core features of the concept are described further below, split up into different two view points: product owner view and web shop view.

Product Owners

• product owners provide their product catalog along with media, availability and other information possibly required by web shop
• they decide where the products are allowed to be incorporated into: theme shops, special shops only, all shops, shops requesting a specific compensation at maximum ....
• they receive orders from an arbitrary web shop for their products and fulfill them
• fulfilling orders might be done in their own name or with a special label of the shop that forwarded the order
• provide the platform with updates on order states as well as product availabilities
• might change the availability of a product between different web shops incorporating the product due to compensation or selling rates in order to max. the numbers

Web Shops

• select and consume products from the platform catalog if they were cleared for it by the owner
• use the platform processes to place orders for products belonging to the product
• use the infrastructure to handle payment processes

There are some more ideas and visions but it would lengthen the blog entry unnecessarily.

Defining and Handling Functional Processes

My last blog entry gave a broad overview on what I am currently after in my sparetime. One core question that plagues me for quite a while now, is on how to define functional processes within the application: incorporate the process definition into the messages or describe it explicitly within the application architecture?

Application Architecture

Describing functional processes and the depending data flow explicitly within the application architecture has two major benefits

• Known structure and data flow eases the pain of extending the application according to customer needs
• Predictable behavior due to known structure and data flow eases the pain of monitoring and managing the application in operation mode

Messages

Incorporating the functional process description into the messages being passed around has one major benefit

• Flexibility of application reconfiguration enables the application management to quickly react to changing requirements

Hence, the main question - as always in computer science - is to answer wether to have a known structure which is easy to monitor and maintain but difficult to extend or to have a highly flexible application which allows the product owner to quickly respond to customer requirements.

In a generic approach, achieving both targets is difficult (or near to impossible), but as I know about the context the application will be deployed within, there is a chance to nail down some basic structural aspects which ease the pains of the operations and support team while maintaining maximum flexibility within that borders.

Therefore the application is structured into so-called domains which handle specific topics, like orders, customers, payments or recommendations. These domains are pre-defined and receive messages dedicated to their specific topic. Within a domain, eg. order, the messages tell the processing components on how to handle them. This allows the product manager to offer max. flexibility to the customer when it comes to the specification of a certain behavior within a domain. This approach is outlined in the iServices wiki.

Finally has to be said, that the messages floating around in the system as well as the provided domain interfaces follow a functional approach instead of a technical one. This particularly refers to the way how these artifacts are designed: they try to map the functional context instead of its technical implementation.

The next blog entry will discuss a concrete application context as I have been too vague on that topic so far :-)

Designing an Asynchronous eCommerce Solution

Ever since I took some very interesting classes about parallel computing (thanks to Dr Clemens Grelck) I am interested in high-performance and low latency computations. Unfortunately, most of the topics back then were academical as the there was only limited access to parallel computers and affordable multi-core (desktop) processors were far from being available.

But nowadays things look a bit different ...

As I am always in search for interesting problems in the ecommerce domain, I recently found myself asking the question: why do most of our products process inbound shop requests in a sequential manner? While executing these calls, precious resources are being blocked since the application awaits a response to its query. So, is there a serious reason not to switch from synch to async non-blocking communication and - beside that - apply some of my previously academical knowledge in a beloved field?

In order to answer that question, I started a little open-source project called iServices which is available on GitHub and actually serves as repo for my own proof-of-concept implementation and further ideas on that topic.

At its core the platform is based on AKKA and provides an ecommerce specific runtime on top it (well, actually it is ecommerce driven, but we will see where the road leads to ;-)). It makes use of functional interfaces and functional messages being passed around. The whole platform is stateless and thus highly scalable to serve large numbers of parallel messages.

One of the major reasons to switch the underlying communication model is to enhance the use of available hardware resources which in turn reduces operational costs compared to sync'ed solutions when it comes to constantly rising number of requests. As the retail market always is in search of ways to save money and increase the margin, this project is a try to proof that the demand can be supported by such an architecture.

But each coin has two sides .... getting a sync'ed application mastered by the developers as well as the application management is much easier than the same applied to an asynchronous solution. The reasons are complex communication structures and information flow behavior that sometimes might seem arbitrary.

The next blog entry will discuss whether to let the platform define functional processes or provide these to the messages and let the platform just react on that.