I think, you shouldn't be. With the shortcomings any ranking system has, there's not much at stake, if you're not faring better with these ranking systems. Ultimately, what matters is the number of visitors you get on your blog, and the numbers you're able to retain with your blog.
As far as Alexa rank is concerned, it changes on a daily basis, so, if someday, you get a visitor count double the other days, you may see the alexa rank fall down too much, and will get back to the usual place in the with the usual traffic on other days. Since alexa ranks your blog or website in comparision to other sites/blogs, the performance of other websites do matter a lot in calculation of your blog's overall ranking, and comparision of your site/blog with others'.
For now, you should be happy that there are only 1,34,673 websites/blogs ranked by alexa which are better than yours, and it's quite an achievement.