Well, yes and no. Sure, you can get hit with an algorithm change or penalty that destroys all your traffic. However, if you have good people who know what they are doing, this is not likely to happen, and if it does, it is easy (in most cases) to get your visits back. Panda and Penguin are another story, but if you get hit by those it is typically not accidental.

Google makes constant efforts to improve the search algorithm and detect quality content and that is why it rolled out the Google Panda 4.0 update.  What is this update really doing? It deranks low quality content and boosts the high quality one, according to Google’s idea of valuable content. In the screenshot below you can see what kind of content you shouldn’t be looking into if you are really determined in making a mess out of your organic traffic. 
Beyond organic and direct traffic, you must understand the difference between all of your traffic sources and how traffic is classified. Most web analytics platforms, like Google Analytics, utilize an algorithm and flow chart based on the referring website or parameters set within the URL that determine the source of traffic. Here is a breakdown of all sources: