I've been giving a lot of thought to this website, and to my Bascule generator which builds this and The Right Notes. The workflow to create a new post isn't great, being tied to the command line installation of the tool on my computer. When new content is created for The Right Notes, my Dad types up his programme in a web interface I've built, the file is committed to github, and then I have to do a git pull, tidy up the source, run the generator, preview it in a local web server (Jetty), and then do an aws s3 sync command to push the new content live. It's a bit slow, and it is in part why neither website has been updated recently.

It would be easier if I could update either website from any computer, through a web application. Improvements could be made to the web interface I built for my Dad, but the real problem is the generation phase. Can I move that to the cloud? One, expensive, solution could be to create an AWS EC2 instance running Windows or Linux, deploy bascule there, and connect to it via RDP. But that's just moving the problem into the cloud, not fixing the problem. I think it's time to re-engineer bascule as a serverless web application, running on AWS Lambda.

To that end I've been experimenting with the Ktor framework, and its serverless wrapper Kotless. Kotless is very immature, and it's hard to know how much it is being actively developed. Like all Jetbrain's projects, its Microsoft Windows support is nearly non-existent. It took me a while to get a working project, including building my own version of the 0.2.0 branch. I know that all sounds like a lot of work and a risky prospect, but I do believe that Ktor is the best approach for me, and Kotless is the only way to make it "serverless". (I'm no longer using Osiris, another Kotlin serverless toolkit, because it's no longer under development.)

I've also been experimenting with writing web pages not in HTML, but in kotlinx.html - a return to server-side rendering. It's working well, though I am not yet convinced it is the right thing to do. It is quite nice being able to write most, if not all, of the project, in Kotlin.

There's some sample code in my KotlessTestBeta project on github, and for now the draft website is accessible at https://kotlessbeta.liamjd.org/.

Making Bascule 'Serverless'

To move bascule to the cloud does mean a complete rewrite. I'm largely OK with this; there is a lot I am not happy with the current build of bascule. I often struggle to remember, or work out, where to start editing the code to change or fix something. Certain features, like the ability to re-order blog posts necessary for my Dad's website, have been difficult to implement. The mechanism for caching (to avoid regenerating unchanged files) is rather complex. Moving to the cloud is an opportunity to revisit some design decisions.

A new architecture is required, and I am thinking something along these lines:

  • An editing web-app is built, from Ktor, Kotless, perhaps kotlinx.html or Jetpack Compose Web. This will be deployed to AWS Lambda via the API Gateway and S3
  • Source markdown files are saved in an S3 bucket
    • No longer use git to store the source files; I'd lose the proper git version control, but much easier to rename files. Use S3 meta-data to store statuses such as "draft", rather than relying on the filename
  • Authentication via AWS Cognito
  • Generation will be triggered through a REST call to a Lambda function, which would scan the S3 bucket source files (markdown, handlebars templates, css) and generate a copy of the site in S3
    • Perhaps this would allow a "preview" function to be built?
  • Publishing would simply be a matter of copying the generated files to the appropriate S3 bucket for the website

If I do decide to go "all in" with Serverless on AWS, what other tools could I use? For instance, my test project KotlessTestBeta treats S3 as a file store, but S3 isn't a file system, and it has taken a bit of work to try to treat it as such. On a recent training course, it was suggested that it's bad practice to use S3 like this - "walking" the S3 bucket is an expensive operation, in terms of time and money, as it requires a lot of requests and data egress. The tutor on the course suggested storing all the S3 object keys somewhere else, like a DynamoDB database. If I do start to use DynamoDB, suddenly there are a lot more options available to me to manage and structure the website.

But this does seem a bit of a betrayal of the goal of bascule, namely that a project would be defined purely by the Markdown source files, the yaml metadata at the top of each file, and a single project definition in yaml. If I do start using a database behind bascule cloud, there is a risk and temptation to turn this into Cinnabar mark 2 (or mark 3?).

I'm conflicted. Not sure which path to go down. And hence, no progress has been made beyond a few experiments.

Oh, and if you were wondering what happened to Project Herschel? It's been abandoned. I realised that I was making myself stressed, and actually quite depressed, trying to do something so far out of my comfort zone without a clear idea of how it was going to work.