https://jefclaes.be/tags/infrastructure/ Recent content in infrastructure on Jef Claes Hugo -- gohugo.io en-us Sun, 12 Aug 2018 14:58:00 +0200 https://jefclaes.be/2018/08/ges-scavenging-and-hidden-cost-of-link.html Sun, 12 Aug 2018 14:58:00 +0200 https://jefclaes.be/2018/08/ges-scavenging-and-hidden-cost-of-link.html Somewhere around a year ago, we started using GES in production as the primary data store of our new loyalty system. The system stores two types of data. External services push batches of dumb downed events to the loyalty system. For example: a user logged on, played a game or participated in a competition. These events are transient by nature. Once the loyalty system has processed them, they only need to be kept around for a few days. https://jefclaes.be/2018/05/amazon-redshift-fundamentals.html Tue, 01 May 2018 14:17:00 +0200 https://jefclaes.be/2018/05/amazon-redshift-fundamentals.html Late 2017, we set out to replace and upgrade our existing reporting and analytics infrastructure with something that would be a better fit for our workloads. Keeping costs and required maintenance at a minimum would be a nice plus, making for an easy sell. After a bit of research, it was obvious Amazon Redshift had the potential to tick all the right boxes. While steadily porting the most problematic workloads away from our existing infrastructure, I started writing an investigative article on the fundamental concepts of Amazon Redshift. https://jefclaes.be/2014/10/programmatically-force-create-new.html Sun, 26 Oct 2014 16:59:00 +0100 https://jefclaes.be/2014/10/programmatically-force-create-new.html I have spent the last week working in an integration test suite that seemed to be taking ages to run its first test. I ran a profiler on the setup, and noticed a few things that were cheap to improve. The first one was how a new LocalDB database was being created. An empty database file was included into the project. When running the setup, this file would replace the existing test database. https://jefclaes.be/2013/12/databases-are-growing-on-me.html Sun, 22 Dec 2013 21:53:00 +0100 https://jefclaes.be/2013/12/databases-are-growing-on-me.html I learned all about logical design of relational databases back in school; tables, columns, data types, views, normalization, constraints, primary keys, foreign keys… At the same time, I learned how to use SQL to put data in, and how to get it out again; INSERT INTO, SELECT, FROM, WHERE, JOIN, GROUP... In the first project I worked on just out of school, we weren’t doing anything interesting with databases; we didn’t have that many users, or that much data. https://jefclaes.be/2013/11/an-event-store-with-optimistic.html Sun, 10 Nov 2013 18:25:00 +0100 https://jefclaes.be/2013/11/an-event-store-with-optimistic.html Like I mentioned last week - after only five posts on the subject - there still are a great deal of event sourcing nuances left to be discovered. My current event store implementation only supports a single user. Due to an aggressive file lock, concurrently accessing an aggregate will throw an exception. Can we allow multiple users to write to and read from an event stream? Also, what can we do about users making changes to the same aggregate; can we somehow detect conflicts and avoid changes to be committed? https://jefclaes.be/2013/10/event-projections.html Sun, 27 Oct 2013 17:43:00 +0100 https://jefclaes.be/2013/10/event-projections.html In my first two posts on event sourcing, I implemented an event sourced aggregate from scratch. After being able to have an aggregate record and play events, I looked at persisting them in an event store. Logically, the next question is: how do I query my aggregates, how do I get my state out? In traditional systems, write and read models are not separated, they are one and the same. Event sourced systems on the other hand have a write model - event streams, and a separate read model. https://jefclaes.be/2013/10/an-event-store.html Sun, 20 Oct 2013 17:30:00 +0200 https://jefclaes.be/2013/10/an-event-store.html Last week, I implemented an event sourced aggregate from scratch. There I learned, that there isn’t much to a naively implemented event sourced aggregate; it should be able to initialize itself from a stream of events, and it should be able to record all the events it raises. public interface IEventSourcedAggregate : IAggregate { void Initialize(EventStream eventStream); EventStream RecordedEvents(); } The question I want to answer today is: how do I persist those event sourced aggregates? https://jefclaes.be/2013/08/eventual-consistent-domain-events-with.html Thu, 15 Aug 2013 14:03:00 +0200 https://jefclaes.be/2013/08/eventual-consistent-domain-events-with.html Working on side projects, I often find myself using RavenDB for storage and IronMQ for queueing. I wrote about that last one before here and here. One project I’m working on right now makes use of domain events. As an example, I’ll use the usual suspect: the BookingConfirmed event. When a booking has been confirmed, I want to notify my customer by sending him an email. I want to avoid that persisting a booking fails because an eventhandler throws - the mail server is unavailable. https://jefclaes.be/2013/03/putting-my-ironmq-experiment-under.html Sun, 17 Mar 2013 16:10:00 +0100 https://jefclaes.be/2013/03/putting-my-ironmq-experiment-under.html Two weeks ago, I shared my first impressions of IronMQ. Last week, I looked at some infrastructure to facilitate pulling from IronMQ. This implementation worked, but I hadn’t put it under stress yet; “First make it work, then make it fast”, and all of that. I arranged a simple scenario for testing: one message type - thus one queue, where there are eight queue consumers that simultaneously pull messages from that queue, and dispatch them to a handler which sleeps for one second. https://jefclaes.be/2013/03/some-experimental-infrastructure-for.html Sun, 10 Mar 2013 17:29:00 +0100 https://jefclaes.be/2013/03/some-experimental-infrastructure-for.html I wrote about using IronMQ as a cloud-based message queue last week. In that post I explained that you can go at using IronMQ in two ways; either you pull from the queue yourself, or you let IronMQ push messages from the queue to your HTTP endpoints. At first sight, the latter allows you to outsource more infrastructure to their side, but upon closer inspection it also introduces other concerns: security, local debugging and scalability. https://jefclaes.be/2013/03/first-ironmq-impressions.html Sun, 03 Mar 2013 16:45:00 +0100 https://jefclaes.be/2013/03/first-ironmq-impressions.html First time I touched messaging was in the first few years of my professional life working on software that supported fire departments in their day-to-day activities. The dispatching software would send messages to a proprietary broker, which in its turn would forward them to interested subscribers; other dispatching clients, or services. To ensure availability, the broker component could failover to a different machine, but that was about it. It didn’t allow you to queue or retry messages; if you weren’t up when the messages were forwarded, you would never receive them. https://jefclaes.be/2012/05/painless-database-logging-with-mongodb.html Sun, 20 May 2012 17:18:00 +0200 https://jefclaes.be/2012/05/painless-database-logging-with-mongodb.html While browsing the source code of the ELMAH mongoDB provider, I learned about a special type of collections: capped collections. From the mongoDB documentation: Capped collections are fixed sized collections that have a very high performance auto-FIFO age-out feature (age out is based on insertion order). In addition, capped collections automatically, with high performance, maintain insertion order for the documents in the collection; this is very powerful for certain use cases such as logging.