It's been a productive 3 years...
A little over 3 years ago I was given the opportunity to step away from my day to day job of doing the usual Microsoft Full Stack(MVC / EF / SQL Server, etc) development with a team for 10 years and jump into the world of Open Source doing NodeJs and MongoDB development as a internal consultant. In my old role we had the typical PM, BA, QA roles and we used Agile. In the new role I assumed all those roles so I had to become a little more organized and brush up on my people skills. I used Trello to help with organizing projects and left my coding seat a few more times a day to talk to my clients to help with my people skills.
Moving from the Microsoft stack to an Open Source stack was a major evolutionary change in how I looked at and created software. It really opened my eyes to what’s out there. My first NodeJs App used Bootstrap, DataTables, Jade, Express, and Mongoose with quite a bit of jQuery glue to keep it all together. It was reliable and the user was happy with it but I didn’t like how my client side code was a little unstructured. I wrote 1 or 2 more apps that way in 2012 before looking for a better way in early 2013. I started looking at Angular.js because I didn’t really fall in love with what I saw in Backbone.js and Ember.js and I liked that Google had backed Angular.js.
After going thru the week or so of getting used to Angular I had the ‘Ah Ha’ moment when I realized I could pass a $scope object from my partial view thru a service to my backend and save it with little or no manipulation(and return the data and populate the page as well). I was sold on Angular.js so I used 'MEAN’ on my next 2 projects. I was writing less code, it was more organized, and I had less bugs.
Along the way, I’ve experimented with many different technologies. One area that I’ve really enjoyed exploring is NoSQL databases. I’ve been doing relational db work using Oracle, Sybase, and SQL Server(and a little MySQL) for nearly 15 years so it has been a nice change of pace working with new data stores. It wasn’t without challenges, though. I remember trying to model my first MongoDB application. I kept wanting to normalize my typical table data into collections and that just doesn’t work. I finally got used to organizing my data structures in MongoDB and now I find it challenging to go back organizing data to 3rd or 4th normal form.
One book that really helped me to understand the differences between NoSQL databases was the Seven Databases in Seven Weeks book(http://pragprog.com/book/rwdata/seven-databases-in-seven-weeks). I highly recommend it. As I stated earlier, my initial foray into NoSQL was MongoDB. MongoDB is the perfect database for someone doing web development in NodeJs. With the MEAN stack, you’re dealing with JSON from the client side all the way back to how the data is stored in MongoDB(BSON). You even write your MongoDB queries in JSON(albeit some of the aggregation framework queries can get really ugly). I was one of the lucky guinea pig’s to take MongoDB’s first classes(https://university.mongodb.com/courses/10gen/M101/2012_Fall/about). I highly recommend take this class as well to really strengthen your understanding of queries(and using the aggregation framework in particular).
As I read more about NoSQL, I started messing around with Redis, LevelDB, and Riak. I used Redis as a session cache in 2 of my NodeJs applications and I also used it as a data cache for a REST api that was used by numerous groups in our company. I experimented with replacing Redis with LevelDB in one application but I couldn’t find any overlying reason to push LevelDB over Redis. I did learn that I could do just about everything in LevelDB using LevelUp that I could do in Redis using LevelUp’s plugin library. I haven’t had an opportunity to use Riak in a production project but I really like how it operates(no master, eventually consistent, easy to setup)
Another area that we’ve spent a fair amount of time on is cloud infrastructure. We really didn’t have a good internal Ubuntu environment to use since our group was primarily doing MS stuff so we experimented with PAAS as well as hosted VM’s. We initially used Heroku and Nodejitsu to deploy our projects. We were pretty happy with each of these services but we needed a little more control of our resources so we started looking at Joyent’s SmartOS VM’s. We really liked running NodeJs on SmartOS but it was a little too costly for the small projects we were working on. Around this time Azure was coming out of beta so we experimented with their PAAS option as well as their hosted VM’s. We realized early on that their product wasn’t quite ready yet. I really liked their UI for deploying and managing VM’s but we saw had some performance issues. We then experimented with Rackspace, Digital Ocean, and Amazon. I don’t have anything bad to say about any of these cloud service. I currently run 2 personal sites on Digital Ocean but the main reason we ended up using Amazon was that we had a corporate account with them.
I can’t say I’ve been disappointed using Amazon’s products, though. We’ve mainly used their ELB, EC2, and S3 products but I hope to have the opportunity to use Cloud Front and Route53 in the not too distant future. I’ve really enjoyed using their NodeJS api. I wrote a small application using their api to back up our MongoDB databases nightly and upload the files to S3. It’s been running for 2 years with pretty much no issues. I’m now working on scripting out a deployment for one application that uses ELB and several EC2 instances. It will create several new instances, add them to the ELB and then slowly retire the older instances as the newer instances come into service.
Yet another area I’ve been messing around lately is search. I’ve used Solr with several Microsoft apps in the past to do search but I wanted to try something a little different so I started using ElasticSearch. From what I’ve done so far, I really like it. With Solr, you would have to understand the structure before indexing it(schema.xml) but with ElasticSearch you can start indexing right away(dynamic mappings). This doesn’t mean that you won’t have to go back and re-index once you realize people are querying your data differently than you thought they were. I’ve had to change my mappings a couple of times to allow users to use auto completion and exact match querying but it’s been pretty easy to make the changes since I didn’t have to edit a schema file. So far I’m pretty excited about using ElasticSearch. I see numerous places where I could replace regex queries I’ve used in MongoDB.
One final group of tools I have been messing around with lately are Packer.io and Docker.io. I started using Packer.io about 9 months ago to mainly build local VM’s to play around with. Lately I’ve started using Packer(along with it’s EC2 builder) to create AMI’s for my EC2 deployments. Why is this cool? I’ve scripted these AMI’s to include my whole production environment including NodeJs and Redis. Not only does it have my production environment pre-installed but when the EC2 instance launches it retrieves my latest code and executes it. So all I have to do is bring new instances online, add them to the load balancer, and remove the old instances once the new instances are “In Service”. I don’t have to login to the instances at all. Docker.io is something I’ve been experimenting with for the last couple of months. It basically will make everything I did with Packer.io obsolete but that’s a good thing With Docker.io, you create containers that contain each piece of your application. So you’ll have a cache container, an app container, and possibly a database or search container. Some of these containers have already been created by the open source community so you may only have to script out your app container. I will make another post about Docker.io as it deserves it’s own title.
So here’s to another 3 years like I just had. It’s been crazy, hectic and frustrating at times but I’ve learned a ton of useful stuff and wouldn’t change a thing.
Thanks to @donwb and @mickeyyawn on twitter for giving me the freedom to crash and burn!