Impetus for an OO Backend

After a long, multi-faceted discussions on CFCDev a few weeks ago,
one of the participants contact me off-list wondering about a sample
app that illustrated some of the concepts I'd mentioned in the
discussion. I don't have one, and while I could make one up, the
implementation is the easy part. It's the reasoning behind doing
things a certain way that is what's important. So what I'm going
to endeavor upon is a quick overview of how I've come to building apps
the way I have. It will undoubtedly be long winded, but I'll try
to be as brief as I can. ;) Hopefully at the end you'll see
three things:
First, OO is hard to do 'right', since 'right' is both subjective and
ever-changing, second, that the 'right' way is always defined by need
and not by how well you use Design Pattern X, and third, that I don't
have any magical OO power, I've just spent the past three years
tuning a single app and building a large body of experience.

Application LayersTo
put the cart before the horse, here's an image that roughly lays out
what I'm going for. At the top we have the Presentation and UI
Controller layers. The first is just HTML, the second is a
framework of your choice (I use FB3 primarily). The dark blue
section, however, is what I'm going to be talking about. And note
that this diagram is about how the tiers work, not necessarily how the
implementation is structured.

First some back-story. I took over an app that was utter spaghetti
code. About 80,000 lines of it. I spent a month converting
everything to Fusebox 3 (the current version at the time), and that
dropped to about 50,000 lines simply from code reuse. (ed.
note: That's a 37% decrease in code size; use a framework people) There was no MVC in the app, it was all single
fuseactions, so the only reuse happened at the fuse level. This
is all on CF4.5, and when CFMX 6.1 came out (and alleviated most of the
bugs and inadequacies in CFMX 6.0's CFC implementation), we upgraded,
and the CFC-ization began in earnest.

My first objective was to stop having
to write so many damn queries, so automated entity persistence was top
of the pile. Fortunately, that's really easy to do, since entity
persistence operations are very closely allied to your database
schema. A few hours of working and I'd built a generator that
would read a table schema from the DB and generate a skeleton BO and
fully implemented DAO for performing persistence operations for
it. It'd also generate a boilerplate factory/manager for the
entity type (getNewUser, getUserById, createUser, deleteUser,
updateUser). With that, I no longer had to have any single-entity
queries in my fuses, rather I request a BO from the appropriate manager
(all of which were singletons in the application scope), do what I
needed, and then call createXXX or updateXXX on the manager with the
modified BO. That saved me enormous amounts of time, particularly
with changes to entity fields (like adding a country to user info),
because I could just regenerate the DAO, and all the persistence
operations were magically updated.

That saved a lot of work, but
it didn't help abstract the business logic out of the UI. My
fbx_Switch.cfm files were still littered with a mix of business logic
and UI processing. So the next step was to start creating service
objects to put the business logic in. This became doubly
important since about this time we started exposing certain
functionality over web services as well as the HTML UI, and that was
starting to lead to enough duplicate logic code to trigger warning
bells. So much of the business logic moved into the service
objects.

An important point to make here is that while managers, BOs, and DAOs are
all pretty much one-to-one-to-one, services are not. An example
would be a permission-based security model, where you have users,
groups, and permissions. That's three distinct manager/BO/DAO
sets, but you probably only need a single SecurityService that deal
with all of them. And you'll probably have a LoggingService that
doesn't care about security, but does care about some aspects of users.

As soon as service objects popped into existence, however, up reared
a major issue with encapsulation. A good issue, mind you, but one that
required creating a solution for. Simply put, the services needed to be
able to talk to each other, but couldn't do that without going to the
application scope to get a reference. The same thing applied to manager
instance as well. The "right" solution, like so many other things, was
driven by time constraints not good design (though it was good enough).
At this time, all managers (and now services) were instantiated into the
application scope directly (i.e. application.securityservice). That was
quickly revised to instantiate them into a managers and services struct
respectively, copy the keys into the application scope, and the pass the
struct into each service (via a setManagers and a setServices method in
the AbstractService superclass). None of the existing app needed to change,
but all the services had references to everything else, and it took all
of ten minutes to put together.

The right solution (from a design perspective) is to use a factory,
and then just pass that factory around. The struct solution is similar
in concept, though not nearly as encapsulated. Needless to say, the
lesson was learned, and factories are now properly used from the start,
rather than being an afterthought.

Ok, time for a breather/cigarette/shot of whiskey. Whew! And here
we go again…

The next problem was that the services often needed to do arbitrary queries
as part of their business logic. This is a two-pronged issue:
SELECT queries, and other queries. And with the SELECT queries,
there are queries that the UI will also need (which still resided in
qry_ files at this point), and others that are strictly needed by the
backend. Along came gateways to solve the first part of the problem.

The set of gateways falls somewhere between the set of services and the
set of BOs in makeup. For example, a UserGateway for users and a
SecurityGateway for groups and permissions. Gateways were also the first
objects to be created via an application-scope factory and use lazy loading
for faster app startup. The methods of the gateways included all the SELECT
statements needed by both the UI and the services, though that may change at
some point in the future.

The problem of the non-SELECT queries was solved by deciding to let the
services perform those queries directly against the DB. It's worth mentioning
that this was only for queries that weren't tied to a single entity; those are
always performed through the entity. Initially, I thought this was a rather
poor way of handling it, but upon further reflection, I've become pretty
comfortable with it. The other solution would be dedicated DB modifier object
that the service delegates to. I'm not sure it's worth the complexity,
unless having all your SQL in SQL-specific objects is important for your app,
because you lose the ability to modify the business logic all in one place.

So now we've got business logic in the services, but we have BOs
(Business Objects, mind you) that are little more than DTOs
(Data Transfer Objects) for easing persistence operations. So entity-specific
business logic was moved into the entities from the services. An example
would be a 'post document' operation. Previously, the DocumentService
would pull the Document BO, set 'isPosted' to true, 'postDate' to now(), etc.,
persist it, and then go on with the other tasks (like distributing
notifications, or clearing the cached list of "recent updates" for the
site). With the new setup, the DocumentService simply calls postDocument()
on the Document BO, which takes care of all the document-related stuff in
one magic step. The service is still in charge of the other non-entity
operations, but all the entity operations are now part of the entity.

So what do we have at the end of it all? A rather complex arrangement of
CFCs (all told, about 200) that makes working with the application much easier.

What other things would I like to see? A real centralized application object
that contains the entire application, and can be passed around instead of the
structs of services and managers for one. This is already partially implemented,
but it's not complete. I'd also like to see user security and logging be
integrated in a more transparent way. Right now, both must be explicitly coded
for, which has led to errors in the past. I'd much rather have that applied
magically by some framework (probably dynamic wrapping of the service objects)
so that it's guaranteed to be consistent across the board.

What about problems? Funny you should ask. ;) Probably the biggest problem
is dealing with DB transactions. CF only exposes the CFTRANSACTION tag, which like
other tags, acts upon it's body. More to the point, there isn't a way to say "start
a transaction, if one isn't already active." Certain business operations
are both standalone operations and part of larger operations, and can be invoked
either way. The solution we've employed is to have both transactional and
non-transactional versions of those methods (the transactional method being nothing
more than a CFTRANSACTION tag wrapping a call to the non-transactional version).
It works, but it's hardly elegant, particularly with methods that take a lot of
arguments.

One solution that I tried out on another app I threw together was to manage
transactions via TransactionManager object that didn't use CFTRANSACTION at all,
but rather use CFQUERY to talk to the database directly. It worked fairly well,
but it depends on an implementation detail of CFMX. Namely that a given request
gets a single DB connection for the duration of the request, and that it is the
ONLY request that has access to the connection. Just for reference BD doesn't
have this feature. It also gets complicated because you must
ensure that your transaction either gets committed or rolled back before the request
finishes, or you'll have some weird issues. However, it does allow you to have
the "start a transaction, if one isn't already active" functionality, which makes
things ENORMOUSLY easier. I'd really love to see this behaviour appear in future
versions of CF, but who knows.

Another problem is that the sheer weight of a complex system can make otherwise
simple tasks fairly daunting. In that case, as long as encapsulation isn't broken,
we simplify. There are a few subsystems that are comprised only of a service object
that contains everything. To come clean, the 'permission' entity from my security
examples above is one. There isn't a permission manager, BO, or DAO, just the methods
in the SecurityService. These are also a major source of the non-SELECT queries that
the service objects need to perform. But like everything else, it's all about picking
the right compromises in each situation. If we need to have a full OO implementation
of permissions, injecting it will be very simple, since the service methods won't
change at all, they'll just stop doing everything themselves and start delegating to
the new backing objects. And that's the real power of encapsulation.

Just for reference, that flat, service-only architecture is usually where I start
with everything because it's quick to develop, and easily facilitates growth down the
road. One good (and publicly known) example is Ray's BlogCFC. It's all in there as
one massive file and that's exactly as it should be, particularly since installation
simplicity is important.

So, after much typing, I'm calling it quits. Hopefully I've met the three
objectives I laid out at the beginning, and haven't caused anyone to shoot smoke out
their ears or pass out on their keyboard. At some point, I may actually show some
code, but I'd like to at least pretend that anyone who gets example code will have
understood the reasons behind why it is the way it is.

11 responses to “Impetus for an OO Backend”

  1. Roger Lancefield

    Hey Barney,

    Great article! Lots of thought provoking stuff and it's comforting to know that others have struggled with the same questions I'm currently dealing with.

    An initial question, what made you decide to stick with FB3 rather than upgrade to FB4, Mach-II, or (these days) Model-Glue?

  2. Barney

    Roger,

    First and foremost, it's what the app's already written with, and none of those other frameworks bring enough to the table to make it worth the switch. I haven't counted in a while, but it's probably about 50 circuits, ranging between 8-10 and 40-50 fuseactions in each one.

    But even if I were writing from scratch, I'd probably still stick with FB3, because the app is so complex but the UI isn't. Even FB4 (which I helped write) is complex to use for simple tasks, and a thin UI layer doesn't warrant that complexity. If you have a UI that requires a lot of page assembly (i.e. complex layouts/portals), then FB4 is a much better choice, but I don't have that need in this particular app.

    It's worth mentioning that when I say "FB3″, I don't really mean your standard run-of-the-mill FB3. I've made a couple tweaks to the core (minor ones, like requiring fbx_Switch in every circuit), but most important is the 'do' UDF, which emulates FB4's DO verb, but without the performance benefits.

  3. Jared Rypka-Hauer

    Hey Barney…

    Good stuff.

    It's interesting to see the differences between FB3, FB4.1, and MG/M2. Since the controller/listener classes in the latter are CFCs, there's many choices in how to use CFCs for the services, logic, and persistence layers. It's the same thing only different, if you get my meaning. ;)

    In the past, I've tended to use a simple composite relationship between DAO/Gateway and the controller… in the controller just doing a cfset thisDAO = createObject(…). For simple things it's still a fine way to go. However I've also been experimenting with more elegant ways to expose the model to the controller, either with something like ColdSpring or ChiliBeans (which is built in to ModelGlue) to generate collection instances that hold the DAO/GW instances. I find myself using collection objects a LOT.

    The thing that's pestering me the most at this point is the overlap of code between some of the read methods in my DAOs (by email, by ID, by email+password) which makes several queries to maintain. I've also been looking at breaking code into several controllers, which can create a code duplication problem (in this case, a good time for inheritance).

    So I appreciate you putting this out here and giving us a point around which to discuss these issues. :)

    One question though… I've read a couple things lately that indicate some people are returning populated user beans from a userDAO.read() method. Personally I think it's messy. Personally I tend to think that calling a UserService, which executes userDAO.read() and then creates a bean and populates it is cleaner and more flexible. Granted there's no "rules" on this sort of implementation-specific situation, but what's your feeling on the issue?

    For my part, right now I have a controller that calls a DAO method it gets from a service and passes that record to it's own private method to generate a user bean from the query. It's not the best and will change, but I like it better than blending factories and CRUD in a DAO.

  4. Barney

    Jared,

    Just to be clear, you have several read() methods in your DAO that take different parameters that can uniquely identify an entity? I have to say I never considered doing it that way, rather having my DAOs be exclusively primary key based, and having lookup methods (in my gateway) that will translate unique identifier X into the corresponding primary key, which is then use for the DAO. It's also a lot easier to generate my DAOs automatically that way, though if you have the unique keys tagged in your database, I suppose you could generate the other read() methods as well. Like I said in my post, the primary impetus for the DAOs was to never again write entity persistance queries, so autogeneratability is of great import.

    For the BO (bean) population stuff, I use a manager (or factory) for this. So I call getUserById() or whatever, which instantiates a user BO, sets the id field on it, and passes it to the DAO to populate (a SELECT query and a bunch of CFSETs). Then the manager returns it to whomever requested it. My DAOs are implementation details of the managers; there's no other way to get a reference to a DAO. Managers handle creation and persistance operations for entities, and that's all they do. They happen to use DAOs to do it. Makes sense to me, but I've been called crazy before. ;)

  5. Mark Mandel

    Barry – fantastic post!

    This actually looks very similar to how I'm currently setting up my applications. So either I now feel like I'm on the right track, or we're both crazy ;o)

    I'd be also interested to know how you setup your CFCs in terms of directories, and how you decided to organise your CFC roots – did you place them all in a directory at the root of your application, or set up a mapping etc?

    I've refactored my current app (my website v2.0) about 3/4 times trying to work out what the best/favourite way to structure CFCs was, and I still can't decide. :o/

  6. Barney

    Mark,

    I'm actually Barney, not Barry, but it's all good. I use a single mapping (/com/domain) that points to the ${approot}/cfcs/com/domain directory, and then branch from there. I usually put my entity objects in a separate package (like com.domain.data), and then put my non-entity objects in other packages. By entity objects, I mean all the per-entity stuff, so BOs, DAOs, managers, and some gateways and services.

    Just for reference, I also have a /ct/domain mapping that points to ${approot}/tags for all my custom tags. So every non-trivial app I build uses two CF mappings, and no external custom tags.

  7. Mark Mandel

    Whoops! Sorry Barney!
    I was obviously having a bad name day.

    Thanks for that – I'm interested to see how differnt people organise their CFCs.

  8. Sam Clement

    I just got round to reading this after bookmarking it long ago. Great post! I always enjoy reading about your insights into coding.

  9. Dharma

    difference b/w DAO and BO

  10. Rob Pilic

    Barney, I just read this post and found it tremendously helpful as I am trying to wrap my head around OO concepts. I am curious if you still use gateways for all your non-entity queries. The way I've read gateways being used is for aggregate queries, and sometimes just to return a query object instead of a collection of bean objects. Also I see you use managers AND services, where managers just manage the beans and services handle all the business logic – the way I've previously read is to have service objects handle both. I guess it's a matter of preference but I wonder if you had any thoughts on it. Again thanks for the great post.

  11. Barney

    Rob,

    Yeah, in general, I prefer to keep my gateways and DAO's separate. So DAO's operate on a single instance of a business entity, and gateways are basically collections of SELECT statements that don't return objects. Most of the time they return recordsets/queries, but some return number or strings (like a getXXXCount style method).

    Regarding services/managers, it's mostly personal preference. I happen to generate my managers automatically, so having the services separate (which are largely hand-written) makes future regeneration a lot easier.

    In the great scheme of things, the specifics don't matter too much. As long as you're encapsulating all the needed functionality in a logical and consistent way, it's hard to really be "wrong".