Having been some time with an in-memory mutable dictionary serving as a data store, the time seemed right to introduce a permanent data store.

Selecting a data store

Choosing a store for a solution is a key architectural decision and requires evaluating a huge set of factors including requirements, domain-store match, cost, scale, read vs write counts, eventual consistency vs availability, transactional integrity, etc.

There are a number of different stores that are reasonable for this project. Ideally I’d try implementing them all, however I don’t expect that to happen in the short-term. These are some options:

  • Document store e.g. MongoDB. This may be appropriate given our aggregates (template and asset) are quite complex objects.
  • Search index e.g. ElasticSearch. If there are many orders of magnitudes more reads than writes then optimizing for querying using a search index may be appropriate.
  • Event store. An event store, typically implemented manually on flat files, records all the business events triggered by API or system actions. It is often used in a CQRS system in concert with a read-specific store. Having read/write separation in our stack allows for this option.
  • Relational database e.g. SQL Server. Provides transactions, is the most mature, and the best supported by libraries.

Choosing between these relies on criteria that don’t exist because this isn’t a commercial project, so because I’m learning and trying to reduce the number of learning variables I chose the tried-and-true, a relational database.

As an aside, as an architect we also want to allow for future contingencies where it is practical. By providing a clear persistence layer in the code and comprehensive integration tests, we allow the choice of store to be re-evaluated in the future (providing the organization is prepared for the cost).


F# has an excellent feature, Type Providers, that generate strongly typed code at compile time for interacting with data sources. This page has a nice comparison of some type providers for F#. For no particular reason I selected SQLProvider

As mentioned earlier, the domain aggregates are relatively deep, so they map to multiple tables, which is why in a production solution I’d lean towards a non-relational store. Here is a comparison of the domain type template, and the tables that store it in SQL (constraints excluded):

type FieldValue = 
    | StringField of string
    | DateField of System.DateTime
    | NumericField of float

type FieldDefinition = {
    Id: System.Guid
    Name: string
    Field: FieldValue

type Template = {
    Id: System.Guid
    Name: string
    Fields: FieldDefinition list
    MaintenanceProgramId: System.Guid option
CREATE TABLE [dbo].[FieldValue](
    [FieldValueId] [uniqueidentifier] NOT NULL,
    [FieldDefinitionId] [uniqueidentifier] NOT NULL,
    [AssetId] [uniqueidentifier] NULL,
    [ValueType] [tinyint] NOT NULL,
    [StringValue] [nvarchar](max) NULL,
    [DateValue] [datetime] NULL,
    [NumericValue] [real] NULL)

CREATE TABLE [dbo].[FieldDefinition](
    [FieldDefinitionId] [uniqueidentifier] NOT NULL,
    [Name] [nvarchar](255) NOT NULL,
    [TemplateId] [uniqueidentifier] NOT NULL)

CREATE TABLE [dbo].[Template](
    [TemplateId] [uniqueidentifier] NOT NULL,
    [Name] [nvarchar](255) NOT NULL,
    [MaintenanceProgramId] [uniqueidentifier] NULL)

The end result is that we end up with some fairly complex queries and mapping logic. One of the sources of complexity is that the FieldValue table is used both for default template field values (when AssetId is null) and asset field values.

The query contains two left outer joins because a template may exist without any fields. This is shown using the rather cryptic (!!) operator. It would be nice if a more pleasant name could be used for this operator.

The mapping between discriminated unions and the underlying store can be handled by joining to a different table for each case, or by the approach used here which is to have a different column for each case. Using different columns makes the query easier, but results in a sparse table. Given we only have three cases the sparseness of the table shouldn’t be a big space penalty.

The read mapping is shown below. The full code is available here.

let private mapSingleTemplate (rows:TemplateQueryResultSet list) : Template =
    let fields = [ for row in rows do
                    let (_, defn, value) = row
                    if defn.FieldDefinitionId <> System.Guid.Empty then // empty guid means template has no fields
                        yield {
                            Id = defn.FieldDefinitionId
                            Name = defn.Name
                            Field = match value.ValueType with
                                    | 1uy -> StringField(value.StringValue.Value)
                                    | 2uy -> DateField(value.DateValue.Value)
                                    | 3uy -> NumericField(float value.NumericValue.Value)
                                    | _ -> failwith "Unknown field type"
    let (templateCols, _, _) = rows.Head
        Id = templateCols.TemplateId
        Name = templateCols.Name
        Fields = fields
        MaintenanceProgramId = templateCols.MaintenanceProgramId

let private templateByIdQuery (dc:DbContext) id : System.Linq.IQueryable =
    query { 
        for template in dc.Dbo.Template do
        // (!!) means left outer join
        for fieldDef in (!!) template.``dbo.FieldDefinition by TemplateId`` do
        for fieldVal in (!!) fieldDef.``dbo.FieldValue by FieldDefinitionId`` do
        where (template.TemplateId = id && fieldVal.AssetId.IsNone)
        select (template, fieldDef, fieldVal)

let private templateById id : Template option =
    let dc:DbContext = Sql.GetDataContext()
    let rows = templateByIdQuery dc id |> Seq.toList
    if [] = rows then None else Some (mapSingleTemplate rows)


In my earlier post on unit testing with xUnit I had difficulty with type conversion that led to some very ugly code. By moving away from .NET Core other, more functional, test libraries became available. The tool topping google search results was FsUnit which provides readable and fluent wrappers for the several major .NET unit testing frameworks.

I believe that parameterized tests are a valuable means of keeping code DRY and separating data from tests. However as my previous post noted, framework support for parameterized tests is very limited once the parameters fall outside the constant expression that can be placed in a .NET attribute. As an extension to existing libraries, using FsUnit means still having to deal with the capabilities of the underlying framework. FsUnit seems to integrate most natively with NUnit which makes NUnit the natural choice.

In the earlier post, there were challenges converting referenced data to and from objects because xUnit expects multiple parameters for one test to be presented as an obj[][], requiring ugly conversions to obj within each data set to allow the test framework to successfully cast the object back to the parameter type. NUnit is more flexible in this respect, requiring only an obj[]. After discarding tuples for readability and, in some cases type conversion, reasons, I found the clearest approach to creating an obj[] was to create a simple record:

type SampleTestNameData = { Tpl: DomainTypes.Template; Expected: Railway.Result<unit,TemplateCommandError> }
let src2 = [|   {
                    Tpl = ...
                    Expected = Railway.Success ()

let SampleTestName (data: SampleTestNameData) =
    let res = Railway.Success ()
    Assert.That(data.Expected, Is.EqualTo(res))

Assert.That fails here because data.Expected has a different generic type to res: Railway.Result<unit,TemplateCommandError> vs Railway.Result<unit>. This can be fixed by replacing Assert.That with Assert.IsTrue((res = data.Expected)) which uses F# structural equality. This is probably not an issue in practice as types would be expected to be more fully specified and therefore match.

At this point we can introduce FsUnit to create nice readable syntax in our test: result |> should equal data.Expected.

Putting it all together looks something like this:

module AmApi.UnitTests.Template

open AmApi
open AmApi.DomainInterfaces
open AmApi.Commands.Template
open NUnit.Framework
open FsUnit

let basicTemplate() : DomainTypes.Template = { 
    Id = System.Guid.NewGuid()
    Name = "templateName"
    Fields = [{ Id =  System.Guid.NewGuid(); Name = "strFieldName"; Field = DomainTypes.StringField("strFieldValue") }] 
    MaintenanceProgramId = System.Guid.Empty 

let emptyRepo = { new ITemplateWriteRepository with
    member this.FindById id = None
    member this.Save (template: DomainTypes.Template) = ()

type CreateTemplateValidationData = { 
    Tpl: DomainTypes.Template; 
    Expected: Railway.Result 
let CreateTemplateValidationData1 = [|  
        Tpl = basicTemplate()
        Expected = Railway.Success()
    };  {
        Tpl = { basicTemplate() with Fields = [] }
        Expected = Railway.Failure (InvalidTemplate "Template may not have an empty list")
    };  {
        Tpl = { basicTemplate() with Id = System.Guid.Empty }
        Expected = Railway.Failure (InvalidTemplate "Template must have an Id")

let ``Create template validation`` (data: CreateTemplateValidationData) =
    let cmd = TemplateCommand.Create(data.Tpl)
    let result : Railway.Result = TemplateCommandHandler.Execute cmd emptyRepo
    result |> should equal data.Expected


Moving to Suave meant a new logger was required as the original logger used the .NET Core LoggerFactory. Generally for logging I’ve used log4net or flat files. In this case I decided to try using the System.Diagnostics.Trace tools – a little old fashioned perhaps – but supported in .NET 4.6.1 as well as .NET Core (future-proofing!).

Creating a trace involves two parts:
1. Creating a TraceSource instance and calling methods on it;
2. Adding listeners to the config file

In this case it is configured to write warnings and higher to the console, and everything to a file log (AmApi.log).

    <trace autoflush="true" />
      <source name="Log" switchValue="All" switchType="System.Diagnostics.SourceSwitch">
          <add name="console" type="System.Diagnostics.ConsoleTraceListener">
            <filter type="System.Diagnostics.EventTypeFilter" initializeData="Warning"/>
          <add name="logToFileListener"/>
          <remove name="Default"/>
      <add name="logToFileListener" type="System.Diagnostics.TextWriterTraceListener" initializeData="AmApi.log" />     

Suave has built in logging capabilities, such as the colored text displayed in the console. It allows these logs to be accessed by creating a logging adapter and configuring it to be used. This is described here, although the interface definition is more advanced than given on that page, as illustrated by the adapter shown below. This implementation calls a method, SuaveLog, on the main logger class that understands and converts Suave log levels to Diagnostics.TraceEventType.

type SuaveLoggerAdapter() =
    let _log (msg:Suave.Logging.Message) = 
        use strWriter = new System.IO.StringWriter()
        let txt = Suave.Logging.TextWriterTarget(Suave.Logging.LogLevel.Verbose, strWriter) :> Suave.Logging.Logger
        txt.logSimple msg
        Logger.SuaveLog (strWriter.ToString()) msg.level

    interface Suave.Logging.Logger with
        member __.logSimple msg = _log msg
        member __.log level msgFactory = _log (msgFactory level)
        member __.logWithAck level msgFactory = async { do _log (msgFactory level) }

Configuration is done in the startWebServer method. I wanted to preserve the existing logging capabilities, particularly the console, so a CombiningTarget was used. CombiningTarget sends the log messages to multiple loggers.

let defaultLog = Suave.Logging.Targets.create Suave.Logging.LogLevel.Info
let logger = Suave.Logging.CombiningTarget([ defaultLog; Util.SuaveLoggerAdapter() ])
let config = { defaultConfig with logger = logger }
startWebServer config handleRequest

To complete the picture the logging class and instance are shown here. I’ve been lazy and used the underscore here to denote the intention to keep the class private. Alternatively an interface could have been created and a private class defined to implement it. A singleton was also considered but one never knows when it might be useful to split logs so I try to avoid that approach.

type _Logger() = 
    let log = new System.Diagnostics.TraceSource("Log")

    let _log (eventType:Diagnostics.TraceEventType) (msg:string) =
        log.TraceEvent(eventType, 0, (sprintf "%s: %s" (DateTime.Now.ToString("yyyy-MM-dd HH:mm:ss.fff K")) msg))

    override this.Finalize() = 

    member this.Info msg = _log Diagnostics.TraceEventType.Information msg
    member this.Warn msg = _log Diagnostics.TraceEventType.Warning msg
    member this.Error msg = _log Diagnostics.TraceEventType.Error msg

    member this.SuaveLog msg (level:Suave.Logging.LogLevel) = 
        let traceEventType = match level with
                                | Suave.Logging.LogLevel.Verbose -> Diagnostics.TraceEventType.Verbose
                                | Suave.Logging.LogLevel.Debug   -> Diagnostics.TraceEventType.Verbose
                                | Suave.Logging.LogLevel.Info    -> Diagnostics.TraceEventType.Information
                                | Suave.Logging.LogLevel.Warn    -> Diagnostics.TraceEventType.Warning
                                | Suave.Logging.LogLevel.Error   -> Diagnostics.TraceEventType.Error
                                | Suave.Logging.LogLevel.Fatal   -> Diagnostics.TraceEventType.Critical
        _log traceEventType msg

let Logger = _Logger()

Suave Gotcha

I had quite the frustrating hour with Suave trying to combine Webparts. In theory it is very simple:

Function1-that-returns-Webpart >=> Function2-that-returns-Webpart

In practice I kept getting an error “Expecting a type supporting the operator ‘>=>’ but given a function type. You may be missing an argument to a function” when trying to run this (thinned out) code:

let getTemplate : Webpart =
    OK "..." 
    >=> Writers.setMimeType "application/json; charset=utf-8"

let route = choose [ pathScan Path.Assets.templateById getTemplate ]
startWebServer config route

The problem – a missing open Suave.Operators – and without it F# didn’t know what >=> was. I guess it was trying to treat >=> as an inline function, but how it goes from there to that error message presently baffles me.


A Change of Direction

I’ve decided to step away from ASP.NET Core for the moment. I’d like to learn more about F# and good functional practice, and from what I’ve learned so far I feel this would be better achieved via tools and frameworks that were designed for F#. Stepping back also allows use of the .NET Framework and therefore my very much missed debugger.

Choosing a Web Framework

In looking at the options for a web framework for F#, two options stood out as being mature and well-supported: WebSharper; and Suave.IO. My choice is to use Suave.IO because I’m building an API and so am not in need of all the client capabilities that WebSharper seems to be strong in. To that end I’ve been through (i.e. typed out) the Suave Music Store tutorial and read through the documentation.

Picking up a new framework is never easy, and less so when the paradigm being used is new. To aid with that I’ve got the Suave source code from github so I can debug into functions to understand what is going on. There were a few problems referencing the linked files (e.g. Utils/Facade.fs) when building it, which seemed to mysteriously evaporate when the project was reopened. So I’m finally at the point where I can ‘port’ my existing code over the Suave.IO and see what happens. This is where it is at so far, but I hope to report more progress in a day or two:

let webPart =
    choose [
        path Path.Assets.template >=> choose [
                PUT >=> OK "Template PUT"
        pathScan Path.Assets.templateById (fun guid -> OK (sprintf "Get Template by Id: %s" guid))

        path Path.Assets.template >=> choose [
                PUT >=> OK "Asset PUT"
        pathScan Path.Assets.assetById (fun guid -> OK (sprintf "Get Asset by Id: %s" guid))
        NOT_FOUND "No handler found"

let main argv = 
    printfn "%A" argv

    let config = { defaultConfig with bindings = [ { scheme = HTTP; socketBinding = { ip = Net.IPAddress.Parse(""); port = 5000us }} ]}

    startWebServer config webPart
    0 // return an integer exit code


Using xUnit

A key goal of the architecture of the functional domain project is that it is unit testable. xUnit.net seems to be the default test tool of choice for .NET Core, so I decided to try it out.

xUnit seems to be an evolution of earlier testing frameworks (see this for a comparison), in particular adding data driven testing.

Trial by xUnit

Unfortunately the documentation for xUnit is poor – while the homepage links to basic attribute use, there is no obvious link to documentation on theories and the various forms of data they take – so using data driven testing requires assembling information from widespread Google searching (which this post may add to) or downloading and figuring out the source code (which I just didn’t feel like this time!). Furthermore at one point I receieved a ‘NotSupportedException’ without a line number or anything else indicating what wasn’t supported, eventually discovering that exception referred to the MemberData attribute.

I’m going to engage in a small rant now. This lack of attention to documentation is a disappointing but fairly expected consequence of being open-source. It’s something that was repeatedly infuriating about working with Node.js, and frankly the opposite of what I expect from something that exists in the eco-bubble of Microsoft, an enterprise framework provider. I don’t know the whole governance structure of the .NET Foundation, but to me .NET and Microsoft go together, so perhaps Microsoft could lend a few technical writers to the cause. And to be clear, no disrespect to the developers – their greatest value is in progressing the platform not spending weeks documenting it. End rant.

Using ClassData

Finally I was able to get the xUnit ClassData attribute working in F#, and from this construct a nice base class so data can be easily created as sub-classes.

open System
open System.Collections
open System.Collections.Generic
open Xunit

type BaseTestData() =
    abstract member data: seq<obj[]>
    interface IEnumerable<obj[]> with 
        member this.GetEnumerator() : IEnumerator<obj[]> = this.data.GetEnumerator()
        member this.GetEnumerator() : IEnumerator = this.data.GetEnumerator() :> IEnumerator

type MyTestData() =
    inherit Util.BaseTestData()
    override this.data = Seq.ofList [[| box 4 |]; [| box 5 |]]

type SomeTestClass() =
    let IsOdd value = Assert.True(value % 2 = 1)

To complete the first test, a little additional type coercion was needed to avoid errors like “Object of type ‘…Railway+Result`2+Failure[System.Object,AssetManagementApi.Commands.Template+TemplateCommandError]’ cannot be converted to type ‘…Railway+Result`2[Microsoft.FSharp.Core.Unit,AssetManagementApi.Commands.Template+TemplateCommandError]’. Note the extra (pointless) conversion at line 7 which is the same type as the argument to the test method.

type TemplateValidationTestData() =
        inherit Util.BaseTestData()
        override this.data = 
            Seq.ofList [
                    ({ Id = System.Guid.NewGuid(); Name = "abc"; Fields = []; MaintenanceProgramId = System.Guid.Empty } : DomainTypes.Template) :> obj
                    (Railway.Failure (InvalidTemplate "Template may not have an empty list")) :> Railway.Result :> obj
type TemplateDomainOperationsTests() =
    let ``Template validation`` (input: DomainTypes.Template) (expected: Railway.Result<unit,TemplateCommandError>) =

        let cmd = TemplateCommand.Create(input)
        let result : Railway.Result = TemplateCommandHandler.Execute cmd mockRepo
        Assert.Equal(result, expected)

Overall this is pretty ugly. It’s incredibly far from ideal that the test data essentially gets its type destroyed because ClassData deals in object arrays. An alternative would be to use InlineData with basic types to populate data but that is very limiting as we can’t use an empty list or discriminated union type as data.

The end result is that I’ll live with theories and ClassData for now, but certainly keep my eye on something better, and something more suited to F#. FsUnit may be better, but currently doesn’t support CoreCLR (although there is a PR open since Jun 1 with changes for that).


One weird behavior I ran into was that a let binding kept returning null during a test, so I put a couple of logging statements in and discovered the root of the module is never run. This makes sense given the test runner is instantiating specific classes – but it is a tricky little trap. Solving the problem is as simple as turning the let (line 1) into a function.

let basicTemplate = { ... }

Logger.info (sprintf "1: %A" basicTemplate) // never called!

type TemplateDomainOperationsTests() =
    let ``Test method description`` =
        Logger.info (sprintf "2: %A" basicTemplate) // returns 

First foray into Computed Expressions

The last day has involved a lot of learning, bringing together concepts which had previously been ‘academic’ with some experience in what I now understand to be specific instances of computed expressions/monads, such as binding and promises in JavaScript and async from C#.

Please note this is based on my best understanding so far, but given my limited experience in the subject it may be somewhere between misleading and outright wrong in places. Nevertheless, I hope the way I arrived at the monad destination might help others.

I started this quest trying to solve an indentation problem:

match IsValid template with
| (false, reason) -> 
    Logger.warn reason
    raise (InvalidTemplate reason)
| _ ->
    match repo.FindById template.Id with
    | Some _ -> 
        Logger.warn (sprintf "Duplicate id %O" template.Id)
        raise (DuplicateId)
    | None ->
        repo.Save template               

There are multiple problems with this snippet:
1. This is horribly unreadable, and the more clauses that might return errors there are the worse it is going to get.
2. To provide early return I’ve used exceptions, which are really just glorified gotos and are not ‘with the grain’ of F#

Searching for solutions to this problem led me to computation expressions – a concept I had not yet reached in Expert F# 4.0.

Computation expressions are syntactic sugar, i.e. an abstraction, for handling continuation functions. A continuation function is something that is called to continue executing something (an algorithm say). It is familiar to me from JavaScript, where we provide continuations as callbacks for async activities like Ajax requests or (in Node.js) IO. When I first worked with Node.js I recall (naively) having the same problem – the callback hell where multiple layers of IO calls descended into a nasty pattern of indentation:

fs.readFile(file1, 'utf8', function (err, data) {  
    fs.writeFile(file2, data, function (err) {

Since then I’ve discovered promises which turn this into a pattern without indentation, but that still involves a lot of functions:

// psuedocode only
.then(function(err, data){
    return asyncWriteFile(...);

C# and TypeScript go a step beyond this by introducing keywords, async and await, that hide the wrapping that turns functions into Tasks/Promises. For instance, in the following C#, the GetIntAsync function signature indicates it returns a task, but the body is written to return an int (return 1) and the caller can directly use the result of the function as shown in the function Intermediate(): return await GetIntAsync(). The Main function shows what the await syntax is hiding.

static async Task<int> GetIntAsync() 
    await Task.Delay(5000); //pretend we're waiting for something external
    return 1;

static async Task<int> Intermediate()
    return await GetIntAsync();

static void Main(string[] args)
    // Equivalent of await, but we can only do it in async methods so it is written out fully here
    Task<int> task = Intermediate();
    int result = task.Result;

Computation expressions provide a general way to alter the flow of a program between expressions (statements). To do this a generic wrapper class is required, and an instance of such a class is called a builder. In the above C# example, the generic Task is like a computation expression wrapper, async/await are the syntactic sugar, and the builder is built into the compiler.

The detail of computation expressions is an extensive subject that I’m not going to go further into here. For that I highly recommend the F# for fun and profit series to give an in-depth understanding. Instead I will highlight how I’ve applied one. The computational expression I implemented to replace the earlier code was based on the Railway Oriented Programming concept and can be viewed in full here. The following are the relevant snippets from that code.

Implementing a Railway Oriented Programming computation expression

The Railway module contains the computation expression implementation. Each block of a railway is expected to run a function that returns a Result. If the result is a failure then the Bind method, which is called by let! and do!, will return failure and no continuation function (the argument f) will be run. If the result is a success, then f will be run, and given the value returned from the previous block. The end result is a computation expression that will process continuation functions until it receives a failure from one of them.

module Railway

type Result<'TSuccess,'TFailure> = 
    | Success of 'TSuccess
    | Failure of 'TFailure

type RailwayBuilder () =
    member this.Bind(x, f) = 
        match x with
        | Success s -> f s
        | Failure f -> Failure f

    member this.Return x = Success x
    member this.ReturnFrom x = x

let railway = new RailwayBuilder()

The next code snippet shows the railway computation expression in use. It is the block from line 30 to 41 preceded by the value railway which was defined in the last line of the Railway module.
Line 31 evaluates the expression IsValid template which returns a Result. Because of the do! syntax, this will then call RailwayBuilder.Bind with x set to the result of IsValid template. If x is Failure then bind returns Failure and no further expressions from the railway block are called. If ‘x’ is Success then bind calls the next expression in the block (i.e. line 33).

module Commands.Template

open Railway
open DomainInterfaces

type TemplateCommand =
    | Create of DomainTypes.Template
    | Update of DomainTypes.Template

type TemplateCommandError =
    | InvalidTemplate of string
    | DuplicateId

type ITemplateCommandHandler =
    abstract Execute: TemplateCommand -> ITemplateWriteRepository -> Result<unit,TemplateCommandError>

let IsValid (template: DomainTypes.Template) =
    match template with
    | { Fields = f } when (Util.isNull f) -> Failure (InvalidTemplate "Template must include a list")
    | { Fields = [] } -> Failure (InvalidTemplate "Template may not have an empty list")
    | { Id = id } when id = System.Guid.Empty -> Failure (InvalidTemplate "Template must have an Id")
    | _ -> Success ()

let TemplateCommandHandler = {
    new ITemplateCommandHandler with
        member this.Execute (cmd: TemplateCommand) (repo: ITemplateWriteRepository) =
            match cmd with

            | Create(template) ->
                railway {
                    do! IsValid template

                    let foundTemplate = repo.FindById template.Id
                    let! isDuplicate =
                        match foundTemplate with 
                        | Some _ -> Failure DuplicateId
                        | None -> Success ()

                    repo.Save template
                    return! Success ()

            | Update(template) -> Success () // Not implemented

At the API level, our controller is now free of exceptions and uses F# much more natively.

let result = TemplateCommandHandler.Execute cmd (new TemplateWriteRepository()) 
match result with
| Failure (err: TemplateCommandError) ->
    Logger.warn (sprintf "TemplateController.Create bad request: %O" err)
    this.BadRequest(err) :> IActionResult 
| Success _ ->
    let url = new UrlActionContext (Controller = "Template", Action = "Get", Values = new RouteValueDictionary(dict [("id", box template.Id)]))
    this.Created((this.Url.Action url), "") :> IActionResult