Overthinking design

My home project is coming along, and I can now get data from my SmartThings hub into my Zato ESB.

So far so good. Now what to do with it? I want to be able to view the data, and that means storing it in a structured way so that I can query it. Sounds like a job for a database, but which one?

This is the point where I start overthinking design…

The default option always used to be a traditional SQL database. It’s still a good choice and would be fine for the volume of data I’ll be storing. But then the thinking starts… will it scale? I’m starting out with temperature stats from my SmartThings sensors, and initial observation tells me that’ll be less than one record per hour, which is trivial. I’d have no problem dealing with all of the data for years.

I will want to extend it, though, and store other stats. Some of those might be coming in every five minutes – I might, for example, want to monitor server stats like CPU and network use. Again, though, for one server that’s still trivial.

But what if I wanted to store those stats for hundreds of servers? What if I was building a system that could scale? Would SQL be the right choice? Would a NoSQL database do a better job? If so, which one? Should I consolidate the data so that older data is less granular? What about RRDTool? I’ve used it before and it’s very efficient but doesn’t handle irregular updates. And what about the GUI? A web framework? Javascript talking to APIs? Should I look at complete solutions rather than just the data storage? InfluxDB sounds interesting, should I try that? Am I reinventing the wheel? Where do I find the wheels and how do I know which are best?

This is the point at which design paralysis sets in. It’s so tempting to generalise the design to something that will handle everything and try to consider every special case and build something that will do everything…

… and end up doing nothing.

The truth is it’s often worth starting simple and building a quick and dirty solution. Often the lessons learned from that will be more valuable than trying to come up with something all singing and all dancing at the first attempt anyway. A throwaway solution acts as a first draft, a way to try ideas and focus your mind on the problem – as long as you’re prepared to actually throw it away. It’s a valid way to explore options, even in an enterprise environment, as long as you’re crystal clear that it’s a temporary solution. Even if you end up using a commercial off the shelf system at least you’ll have a much better idea of the criteria you should use to evaluate products.

So with that, I guess there are no excuses… I need to crack on and actually write some code!

in Random Musings

Add a Comment

Your email address will not be published. All comments will be reviewed.

This site uses Akismet to reduce spam. Learn how your comment data is processed.

Related Posts