Migration to Acid-State

Over the last week I've been bed-ridden with a chest infection. Although, on Wednesday I did manage to migrate this site from using persistent and MongoDB over to using David Himmelstrup's acid-state.

Why Not Use *-SQL?

I've never liked using the standard database solutions like MySQL or Postgresql. As many of you are aware, there is often an "impedence mismatch" between the database and the language used to write the software. The phrase "impedence mismatch" is borrowed from electronics and is used to describe the situation where the types and structures used in the programming language do not match up with those provided by the database.

I felt this mismatch even more strongly when I moved over to Haskell (from C#, amusingly). Most of the structures I use in Haskell programs do not correlate very well at all to the table-based structure of modern databases. Additionally, the widespread use of sum-types makes the representation of data in fixed columns quite tedious.

The first large-scale project that I wrote using Haskell had quite a complex data model. Unfortunately, storing the record types in a standard relational database was not a possibility. So, I moved over to using MongoDB. To make life a little easier, I needed a persistent-like abstraction that supported sum-types and a few more complex types (such as Data.Map). So, to that effect I developed the now horribly out-dated mt-mongodb library.

acid-state

Now, just before I wrote mt-mongodb, I initially considered using acid-state. Here was a data store written in Haskell that provided ACID guarantees and let me use Haskell data types. The only reason I didn't end up using acid-state was that we had to be able to provide access to the data-store to other services. These days I'd just give them a nice JSON API and a simple PHP library, keeping the database to myself.

Now, I've been keeping an eye on acid-state ever since then. When it broke away from Happstack and became a standalone library I got even more interested, as that meant it would be a lot easier to integrate into other projects. Now that acid-state has reached version 0.6.5, I figured I'd give it another go.

I was happy to see that the acid-state API hasn't changed all that much, and is just as easy to use. Some looking around also showed that the safe-copy package is the default serialisation path used. This means that it is easy to provide backward compatability between versions of the data structures. It also seems that the plans for Happstack 8 include support for replication and sharding!

In order to see how easy it would be to integrate acid-state with a Yesod application, I decided to migrate this blog.

Data Model, State and Events

The first step in migrating the blog was to migrate the data model from the one generated by the mkPersist function to one using Haskell records. As this blog is very simple, it just meant the definition of User and Post record types and their associated IDs. The IDs are stored as Integers and wrapped in newtypes for convenience. As an example, here is the User data type and the related UserId:

newtype UserId = UserId { unUserId ∷ Integer }
               deriving (Eq, Ord, Data, Enum, Typeable, SafeCopy)

data User = User { userId       ∷ UserId
                 , userName     ∷ Text
                 , userPassword ∷ Text
                 }
          deriving (Eq, Ord, Data, Typeable)

To make these structures work with acid-state we need them to be instances of the SafeCopy type class. We derive the instance of this type class using the deriveSafeCopy function:

deriveSafeCopy 0 'base ''User

Finally we want to define our actual state that we want acid-state to manage. This is defined in the MeadowState data type:

data MeadowState = MeadowState { meadowNextPostId ∷ PostId
                               , meadowNextUserId ∷ UserId
                               , meadowPosts      ∷ IxSet Post
                               , meadowUsers      ∷ IxSet User
                               }
                 deriving (Data, Typeable)

deriveSafeCopy 0 'base ''MeadowState

With the site's state defined, I then moved onto defining the update and query events that I wanted to be able to apply to it. As a first example, here is an Update event to update the Post in the meadowPosts set based on the (hopefully unique) postId field:

updatePost ∷ Post → Update MeadowState ()
updatePost updatedPost = do
  meadow ← get
  put meadow { meadowPosts = updateIx (postId updatedPost) updatedPost (meadowPosts meadow) }

Here is an example of a Query event that tries to find an element of the meadowPosts where the postLinkName matches the argument:

postByLinkName ∷ Text → Query MeadowState (Maybe Post)
postByLinkName linkName = do
  meadow ← ask
  return $ getOne $ (meadowPosts meadow) @= LinkName linkName

Once all the events were written, the acid-state events then needed to be generated for these functions using the makeAcidic function:

makeAcidic ''MeadowState [ 'postByLinkName
                         , 'updatePost
                         , ... ]

Once that was done, I was ready to move onto replacing the use of persistent in the rest of the site.

Foundation, Application and Handlers

First of all, I removed the fields related to the Persistent configuration and the pool of connections to the database from my foundation data type in my Foundation.hs source file. I then added a getMeadowState field with the type AcidState MeadowState from my model definition. To make life using acid-state in Yesod handlers I also wrote two simple functions to run acid queries and updates:

acidQuery ∷ (QueryEvent event, MethodState event ~ MeadowState)
          ⇒ event → GHandler sub Meadowstalk (EventResult event)
acidQuery q = do
  state ← getMeadowState <$> getYesod
  liftIO $ query state q

acidUpdate ∷ (UpdateEvent event, MethodState event ~ MeadowState)
           ⇒ event → GHandler sub Meadowstalk (EventResult event)
acidUpdate q = do
  state ← getMeadowState <$> getYesod
  liftIO $ update state q

With this in place, I was then able to change the makeFoundation function in my Application.hs to use the openLocalStateFrom function to open the local state from a set location on the hard drive.

Now I was able to go round and change various route handlers to use acid-state rather than persistent. As an example, the root resource for meadowstalk displays a list of the published posts. This list is generated by fetching all the posts and passing them to the renderPostList function:

getRootR ∷ MeadowstalkHandler RepHtml
getRootR = do
--  posts ← renderPostList =<< (runDB $ selectList [] [])
  posts ← renderPostList =<< (acidQuery $ PostsByState StatePublished)
  defaultLayout $ do
    setTitle "Home"
    $(lessFile "templates/post.less")
    $(widgetFile "root")

Deployment Gotcha and Conclusion

The only hicough came when deploying the new version of the site to the server using Michael Snoyman's excellent keter: as everyone knows, Keter wipes the directory into which it installs a bundled web-application every time you re-deploy it. This, of course, wipes out the state directory created by acid-state, somewhat undermining it's entire purpose.

To combat this problem, I store the state in the directory /var/lib/meadowstalk and use the openLocalStateFrom function to load the state from there. This means that no matter how many times I re-deploy the application it still retains it's state.

In conclusion, the change over to acid-state was utterly painless, and now I have a completely native data model. I can use Haskell data structures and write all my queries and updates directly in Haskell. It also means that I can drop MongoDB altogether on the server on which this blog runs. I've also gained some new cryptic buttons in my administration interface (shown right), which is always nice.

At some point later in the year (probably after I've got this mail server finished), I have another possble web-project. I must admit, I'm seriously considering using acid-state for that project too.

If you want to know more about acid-state, there is an awesome tutorial over on Happstack's site in their Crash Course.