Of cause there can be many reasons why your site would experience a sudden drop in its rankings, this article will only point to some of them – in the end it will still be up to you to figure out what is going on.
A sudden drop in rankings does not necessarily mean that your site was penalized by Google, it could be that Google changed something in their search engines, changes in how they rank pages can occur pretty often. These changes are mostly minor, and only few sites will notice them – sometimes however, they might apply changes that really stand out, which presumably was the case with the April 19 Panda update, after which a lot of sites started to report a drop in their rankings for different keywords.
In a lot of cases however, you won't be able to know what the problem is, the best thing to do in these cases, is to move on, continue to improve your site and its content, etc.
Doing what Google refers to as over-optimization might lead to ranking problems for your site. This is not what would fit well into the normal category of SEO – which would be improving your sites content and internal linking. This would belong in the blackhat category, and includes things such as keyword stuffing, automatic content generation, etc.
In the case of the drop in rankings of April, this is not likely to be the case, because Google did not release their Over-optimization penalty until April 24.
Having to many pages with to little content
In the case of the April 19 update, the drop you may have experienced could be due to having to many pages with little or no content, aka thin content, which can drag a site down in the search results.
Keep in mind that such problems likely can be reversed trough a higher Site Authority – and how is Authority gained? Likely trough a sites content and its PageRank, where the last part is gained from a combination of content and links.
Links pointing to you
Links pointing to your site is very unlikely to have any harmful effects. There have been reports of people who thought they had a effect, and when having a closer look, it turned out it was a problem with their site anyway.
The main reason why such links can not harm your site, is that it would be to easy for competitors to down your site. Any links which gets targeted as spam, is much more likely to simply get silently devalued.