React/Redux coming from CQRS/ES

Looking at React + Redux, there is a noticeable similarity to CQRS + ES. These front-end and back-end concepts being aligned may be helpful for those that cross the boundary between front and back. However, there are some subtle differences that make the concepts "not quite" fit. Let's explore that.


Actions are Events

This can't be any more explicit than what is stated in the Redux documentation.

Actions describe the fact that something happened

http://redux.js.org/docs/basics/Reducers.html

Action Creators are Commands... and Their Handlers

The command and the command handler are squashed into the same concept. The "command" doesn't travel outside the application, so there's less need to convert it to a distinct command message. In fact, doing so feels awkward and redundant due to the next point.

An event (aka action) is almost always generated by a handler. One of the main reasons commands can fail on the back-end is because they are not trusted, so the code inside the handler must validate the command (aka protect invariants). On the front end, the command is considered trusted because the components + state are protecting the invariants. For example, you won't be able to issue a command (the button will be disabled) if the invariant is violated. (If you can, it's considered a bug.)

There's also how errors are handled. They are typically considered events on the UI (user needs to be notified), whereas command handlers typically just return errors as responses without affecting the domain.

What remains the same about the command handler between front- and back-ends is that the handler manages dependencies for the use case. On the front end that often takes the form of wrangling HTTP API calls.

Store is a View Database, Reducers are View Updaters

This is evident from the note on Reducers.

As your app grows, instead of adding stores, you split the root reducer into smaller reducers independently operating on the different parts of the state tree.

http://redux.js.org/docs/api/Store.html in A Note for Flux Users

Essentially, different properties off of the Store's state represent different views. A reducer is responsible for updating its own view.


But keep in mind...

I'm just getting started with React/Redux. These are mental models based on an understanding of CQRS/ES. "All models are wrong. Some of them are useful." (George E. P. Box) This doesn't map all the odds and ends from CQRS/ES and friends, but hopefully it's useful to you.

ID Generation Strategies

The theories

Historically, common practice is to let the database decide the next ID for a new entity. The database is THE place where consistency was maintained and therefore the best source of truth for what's available next.

Things can go wonky from there when you get into systems with eventual consistency between writes and reads. As a result, I have been investigating other ID generation strategies.

One that has become popular is to use UUIDs for entity IDs. Clients can generate UUIDs at will and they are reasonably guaranteed to be unique. Initially, I thought I would switch to this, but further research brought up 2 problems. One, various platforms have differing capabilities at generating UUIDs, especially looking at HTML5. More importantly, GUIDs are hard to remember, to communicate, to type. Even if they aren't exposed to the users, *I* still have to deal with them when debugging, querying, etc.

So the next idea was to use an ID issuance service, where the client first requests a new ID. This can be retried as necessary until one is obtained. Once obtained, the client makes requests against the ID. The downside here is the risk of unused IDs due to transient failure. I could also imagine misbehaving retries creating swathes of unused IDs.

Then you can diverge into ID request tracking -- clients send a UUID, then poll another resource by UUID to watch for the ID to be generated... or at the very least the UUID can enable retries on creating the same ID. Or you can use a HiLo type algorithm so the client itself reserves a block of IDs ahead of time that it can use until it runs out. Again, with non-sequential IDs (across different clients) and (blocks of) unused IDs as minor drawbacks.

An Implementation

My first attempt at ID generation used the conventional model of creating an ID when an entity is created, based on the next available ID from the database. Then returning that ID to the caller (e.g. a Location header for REST-oriented folks). The client is also sending me a GUID (or UUID) with the request so I can trap retries (... or can I?). Ultimately the next important decision point was:

What do I do if a client sends me the same Request UUID more than once? Then to my horror, the answer is predictably: "That depends." If the client initially failed to receive a response and is retrying the entity creation, (and assuming the server can query generated IDs by UUID), then the correct response would be give them back the same previously generated ID for that UUID. However, if it's months later and due to bad pseudo-RNG the same UUID is generated for a fresh request, then the correct response would be to error. Or perhaps better, I could still return the previously generated ID and depend on the POST itself to fail because the entity had already been created. This is essentially a concurrency error where I expected entity version -1 (not created), but actual version was >= 0.

Implementing ID generation tracking in order to allow at most once creation semantics has its own challenges. Tracking means storing the request ID/entity ID and then either querying or caching this store. Querying can be a problem depending on your ID store. I'm using EventStore so as to avoid adding another database into the mix, and querying each time is essentially like a table scan -- not ideal. Caching is a brand of musical chairs that can work, but can get complicated if "done right". The sticking point on caching for me is expiring least-recently-used entity ID sets (each entity has its own incrementing set of IDs) so that memory usage doesn't grow unbounded. Maybe in 5 years every RequestID/EntityID fits in memory just fine, but maybe it doesn't! Cache miss loading performance is also likely going to be an issue over time.

Current Conclusions

Should I try to guarantee at-most-once creation? Well the record-keeping requirements on the back-end certainly makes me question that!

For a separate ID issuance service, is issuing an ID that never gets used really so bad? (Let's say due to a transient network failure and retry) Having gaps in ID issuance appears to be psychologically damaging to certain personality types, and it *is* good to be conscientious about things "falling through the cracks". What can we do to sate that? I suppose a report could suffice to satisfy that curiosity. Maybe even a follow-up procedure to officially tombstone issued but unused IDs after a certain time period if it's important that every one be accounted for.

That actually wouldn't be procedurally much different from allowing a client to double-post the same customer, causing two different customer IDs with the same data, and then having to administratively go back and delete one. Although the issued-but-unused-ID method leaves your data in potentially better shape in the interim.

For the ID issuance service, there is still an itch that I want to scratch: misbehaving clients (let's say a bug) requesting lots of IDs. You could just say "Who cares if 10 million IDs were issued but unused before we fixed the problem?" and go on about your day. But in principle, I'm unlikely to say that. And I haven't thought through options to determine a good remedy. A throttle is the immediate answer to mind, but that's terribly boring and makes me think that I really need to look at the problem differently to illuminate better options.

Oh, and by the way...

And finally, I want to say that exploring ID generation has brought up a very important shift in the line of thinking about IDs -- that is, an ID is metadata about an object rather than part of its content. When you look at most existing code, you see the entity ID being part of the entity itself. But ID issuance begs to differ because the ID must be known ahead of time before the entity can even be created. The client-generated GUID (which is also ID issuance, but from the client instead of server) also takes this tact. In fact, the SQL database itself also takes this tact in that it has to know what the next available ID is before it inserts the data. But our code has been so tightly integrated with SQL implementation details up to now that it was taken as a given that ID needs to be part of the data itself. When really, that is just what SQL requires - ID being part of the data row.

Functionally, the data itself doesn't usually give a flying rip about its own ID number when actually doing work. The important part is that the infrastructure knows about identity and can appropriately provision work and load data when given an ID. NOTE: In the relational DB world, an entity may care about the ID number of a *separate* entity insofar as it needs to ask the infrastructure to load that other entity's data.

Anyway, this realization has affected the how I model entities (e.g. in DDD, no more aggregate IDs on the aggregates themselves), and so I thought it was worth mentioning.

Install EventStore as a service in Windows

Install Chocolatey

(optional, you can manually download NSSM)

Start a command prompt as administrator and run (copied from the website):

@powershell -NoProfile -ExecutionPolicy unrestricted -Command "iex ((new-object net.webclient).DownloadString('https://chocolatey.org/install.ps1'))" && SET PATH=%PATH%;%ALLUSERSPROFILE%\chocolatey\bin

Install NSSM

From command prompt:

choco install NSSM

Download EventStore

http://geteventstore.com/downloads/

Get Event Store for Windows

Unzip and copy to C:\EventStore (or a location you prefer)

Note: EventStore is available from Chocolatey as well, but as of this writing it wouldn't install for me, and it’s slightly behind current version.

Install EventStore as a service

From command prompt:

nssm install EventStore C:\EventStore\EventStore.ClusterNode.exe --db ./db --log ./logs

Note: This uses the default ports and runs single node. The db and logs directories will be automatically created on startup if they don't exist.

My Next Software Architecture

I've been thinking more about how to architect my next web-based software project. I'll be honest with you, I'm not a pro at this yet, but I'm trying to figure it out and get some experience. So I'm going to bounce some ideas off of you, Internet, as well as work out some of my thought processes. Here are some choices that I have in mind. Note that these things have been around a while, and I've played with them a little, but it's fairly new to me.

Profile and Strategy

Non-distributed

Pretty much all of the web apps I write manage a business's internal workings and are not distributed (in the large-scale sense), or are at most distributed to a few satellite locations. (This has historically been handled by file replication, database replication, and private networks between locations.) Therefore, I'm not going to add extra trappings that require a lot of configuration or overhead. For example, I won't be using a durable message queue. The core of the system will be running in-memory, and the components will mostly communicate in-memory.

CQRS

Earlier in my career, I thought it was a good idea to directly use business entities for UI views. That ends up leading to your domain objects being bloated with some UI-only concerns and vice versa. The common alternative is to create different view models as projections of your live business entities. But in order to do that, you have to load your business objects and map them to view objects. So this ends up with a lot of mapping code maintenance, and the mappings can get complicated.

The CQRS strategy keeps two sets of data updated; the domain (business) object data, and the view model data. The benefit here is that each can evolve at their own pace without greatly affecting the other. It's also a bit faster because it is no longer necessary to load the domain object first -- you just load the data straight from database to client. The downside is that you have to update 2 (or more) sets of data. But overall, it eases the complexity of trying to use one set of object for 2 distinct purposes.

DDD

My problem domains tend to be, on average, moderately complex because I'm representing internal processes of a business. DDD is meant to address complexity, but it's more about behaviors and communication patterns than code patterns. Probably the one code pattern to take away is to structure the domain objects in the same way and with the same names that customers use to describe their processes. That typically means insulating the domain objects from view and persistence concerns, and let it focus on business. If not doing CQRS also, you end up with the mapping code issue.

Messaging

Using messaging brings additional overhead to the project. (As compared to direct method calls on domain objects.) But it decouples the clients from the domain and generally just bring options to the table. In my case, I'm leaning more towards a completely HTML/Javascript-based UI, so direct method calls into .NET are not an option anyway. I could write MVC actions which are coupled to domain methods, but in that case it's about as much overhead to implement as messaging (just a different kind of overhead), and the coupling still causes ripple effects and interface maintenance.

Specific Tactics/Tech

Event Sourcing

This tactic lets you represent your domain objects in persistent storage as a series of events, which are basically just classes with the appropriate contextual information. Examples: CustomerCreatedEvent, CustomerAddressCorrectedEvent, CustomerCreditLineRequestedEvent, etc. You can imagine all of these events having a customer id. The address event, you can imagine having properties related to address information.

This has a number of advantages including trace-ability, replay-ability, easy and performant persistence story. The main disadvantage I see to this is that some of the messaging concerns (events, specifically) end up leaking into your domain logic. However, anything message-related done by your domain objects is usually very simple (plain assignment statements).

But event sourcing also opens the possibility of using non-relational databases. For me, it is usually the case that the domain deals with relationships, and view models are relatively flat. When the domain is event sourced, the persistence format is flat. This opens up the doors to alternative databases (such as NoSQL) which tend to be faster and easier to work with. Which leads me to my next point...

NoSQL Database

A traditional SQL database provides a lot of capability for reporting. However, it's a real pain to work with for application data. Most of us don't think about it because it's become second nature. Running a basic SQL statement requires you to 1) have a magic string somewhere with the query or stored procedure to run, 2) wrangle objects like SqlDataAdapter, SqlConnection, etc., 3) map parameters into the query, and 4) try to run the query and interpret the results. (For select queries, there's the additional pain of mapping/casting the DataSet back to an object.). This is painful enough that most of us create abstractions around this process. The first evolution is a DAL that maps method calls to SQL statements. Later evolutions end up being a repository and/or ORM. An ORM requires that you stay abreast of the ORM-specific extensions and code around its design. A repository (or even a simple DAL) requires manual coding and maintenance. In the end, no matter what you do, dealing with SQL is a lot of work. Contrast that to the potential ease of using NoSQL databases, which could take your object as-is and store it straight to the database (e.g. db.Store(message);). That's with no custom-built abstractions, no magic string SQL statements, and no extra ORM framework to learn. That's one compelling persistence story. Even if you need a SQL database for reporting, this can just be an additional integration point as though it were another read model to update (from the CQRS story). :)

Additionally, some NoSQL databases have REST APIs, which means I wouldn't even have to implement a read-layer for the HTML5 UI. The only part that bothers me about the REST API is security. And I haven't yet researched my options there.

WebSockets
The websockets feature is one of the most exciting web technologies to come out in a while. It allows the server and client (e.g. browser) to push messages to each other in a low-impact way. Previously, I wasn't using any external bus (e.g. MSMQ), because the overhead and administration needed to use it wasn't worth it. But websockets are rather simple to get running and are still a developer concern (as opposed to an administrative concern like MSMQ). I want to use websocket connections to serve as my program's contact with the outside world. It's not a durable bus, but since my program is in-memory and not distributed, that really doesn't matter.

One other interesting point is that web sockets provide better asynchronous capabilities. If I send an AJAX request from JavaScript, to an MVC Action, both the AJAX request (on a separate browser thread) and the MVC Action are kept open and waiting until the program finishes the action. Using websockets, I have the capability to take the command, hand it off to a queue, then have a callback send a message back to that client notifying them of completion. I can also have an event websocket to allow external listeners.

The downside to WebSockets is the feature is not widely supported currently. To host WebSockets natively in IIS, you currently have to have Windows 8 or Windows 2012 Server. As far as clients, WebSockets is not supported on most browser versions aside from the current crop.

HTML5/Javascript UI w/ MVVM
HTML5/Javascript is pretty much the direction that the web has taken. I ruled out both WebForms and MVC (the design, not the project type) for the UI due to both performance and knowledge dependencies. Both approaches (traditionally) post back to the server, let the server make some UI decisions, and then send the decision (or command) on to the business layer (or domain). It's basically an extra hop (to use a network term) as compared to a purely in-browser UI. And as we all know, external communication is often the most expensive part of a given operation.

But performance alone is not enough, and in fact using a solely HTML5/Javascript UI is only possible due to some nice Javascript frameworks. My personal choice in the matter is jQuery and Kendo UI (which has controls, MVVM, data sources, etc.). With MVVM, you can completely separate the view from the model, which makes working with the code a lot easier. I end up with the following for each view: view (.html), style (.css), view helper (.js, for any extra control-related functions the view might need), and view model (.js). Then for more complex scenarios, I add more view models and/or script files which listen for view model changes and react accordingly. It's all pretty fast.

This style of development does take some getting-used-to compared to server-side UI development. But one of the reasons I like Kendo UI is because there are a lot of functions that are pre-integrated as compared to taking separate libraries for UI controls, MVVM, validation, etc. and trying to integrate them together.