I've been thinking about it for a while and after I installed Ghost on localhost and took a look at the writing interface, I was immediately sold.
It's not a move for everyone, since as a blogging platform, Ghost is still way behind Wordpress in terms of features but if you want to simply focus on the writing, it's one of the best ways to go.
Another important factor in this decision was DigitalOcean (just so you know, this is a refferal link). I had an account, close to 20$ in credit and my mind set on slowly moving all my sites there.
After a couple of experiments and close to 3 or 4 droplets destroyed, I finally got it up and running with the help of this very helpful tutorial from ghosted.co, taillored specifically for those who want to run Ghost on DigitalOcean. It was a very educational experience and helpful for everyone, not just DO clients. However, if you don't feel like it, you can easily make a droplet with Ghost.
Porting the posts was a bit more difficult, but I already had mine in markdown format, after a former Wintersmith experiment, so it was reduced to copy / paste, more or less.
In theory, I should have been able to use the HTML version, but I used some search and replace to convert parts of the synatax in the Markdown format preffered by Ghost.
Summing up, I'm glad I did this switch. I feel like Node.js is the future and that Ghost will get bigger in the future. On the other hand, while Wordpress is a great CMS, it feels way to overkill for a simple blogging platform.
So Ghost it is!