When you set your eyes on a target, you have established the point of aim (POA). From the moment you start to pull the trigger, hundreds of variables will affect the trajectory of the projectile until it ultimately hits its destination. Sample size is also relevant here - when evaluating a rifle's accuracy, it is common to shoot no less than 3 but rarely more than 10 shots into a target, and then measure the "spread" between them. Now, the place where an individual round hits, is the point of impact (POI). Pursuing accuracy is best expressed as the study of reducing the delta (difference) between the POA and POI. The unit of measure commonly used to describe this delta, is the Minute Of Angle, or MOA. 1 MOA is a 1.047" circle at 100 yards, 2 times that at 200 yards (2.096"), and so on in a linear progression. It is important to understand that this delta is actually the product of several factors, each of which has contributed to the size of the delta. A rifle may produce a 3MOA grouping, but when adding a 7MOA shooter into the mix, the rifle's inaccuracy is hardly the smallest concern.
So, the key concept to absorb is that nothing is inherently accurate, merely less inaccurate. Because this delta is a "stack" of tolerances, accuracy can't be a concrete factor found somewhere along the line - it's a matter of finding the factors which are presenting the greatest cumulative error, and adjusting them to the point where the cumulative error is minimized.
Accuracy Issues, from Aim to Impact
Here are the factors, going from shooter to target, which play the greatest role in reducing accuracy.
Shooter error: This factor and the last factor (environment) are not parts of the weapon system, but are worth mentioning. In a huge number of cases, shooters will spend plenty of money to improve their weapon's performance without improving their own.
Sight Adjustment: It doesn't matter if your sights are on the target if your barrel is pointing 3MOA away from it. Your bullets will follow the barrel, not your sights.
Sight Consistency: Low-end scopes and sights are frequently loose-fitting, and can shift position between shots. It is important that your targeting system, whatever it is, be consistent above all else. The dust cover scope mounts frequently installed on SKS rifles are a prime example of inherently inconsistent sighting systems. Without plenty of tweaking, a dust cover mount will tend to wobble freely over a several-MOA range.
The weapon's action: Certain actions are inherently more or less accurate than others, simply as a function of design tolerances. The AK-47 action, for instance, is optimized to provide excellent reliability through generous internal tolerances - it will function very well even when extensively fouled. However, the loose tolerances also mean that its action is not inherently accurate. The AR-15 is on the opposite end of the spectrum - tight tolerances result in poor performance when fouled, but significantly greater inherent accuracy in ideal conditions.
Barrel condition: A "shot out", eroded barrel will contribute significantly to the variance and inaccuracy of a rifle. The same goes for a barrel which has been extensively fouled.
Barrel temperature: The accuracy of a barrel varies according to its temperature. Any given barrel will start out around room temperature, and heat up as fired, then reduce temperature as it cools.
Barrel / Ammunition Compatibility: This is one of the more interesting points to ponder. The twist rate of the barrel (see Lands and Grooves for explanation) affects how much spin is imparted to the projectile travelling down it. A twist rate which is adequate for one weight of bullet may not be adequate for another. In an M4, the 1:7" twist rate will work optimally with 77gr bullets, but may actually tear apart light varmint bullets. At the same time, a 1:12" twist rate (common on old military-profile barrels) won't impart enough spin to a 77gr bullet, and as a result it will commence tumbling shortly after leaving the barrel.
Ammunition Consistency: Ammunition should be chosen with the rifle in mind. Military surplus ammunition is generally of decent quality; discount ammunition is usually considerably less. The best ammunition is typically a well-tailored hand load made by you, specifically for your rifle. That having been said, the primary causes of inaccuracy in ammunition are inconsistent powder charges, imperfectly formed bullets, and inconsistent case neck crimp.
Bullet Weight Distribution: Small variances in a bullet's shape or density can affect accuracy negatively. A tiny bubble in the lead which the bullet is made from will result in one side being heavier than the other, and in a spinning bullet that will rapidly cause it to go off-center.
Environmental Factors: Environmental concerns are getting out of the area of the weapon, and into the realm of the shot itself. Shooting in a mountainous area will frequently subject you to winds that vary in intensity on a minute by minute basis. A hot day may have visual distortions due to heat, and a desert area will feature an updraft due to hot air rising off of the ground.
Wind: Wind gets its own category. The longer a bullet is in the air, the more time wind has to act against it. Therefore, between a faster bullet and a slower bullet of the same weight, the faster bullet will be less affected by the wind. However, generally speaking a lighter bullet is more affected by the wind than a heavier bullet. This is because it requires a more forceful wind to push a heavier object, than a lighter one. There is limited wriggle room in the last statement for aerodynamics, but as a generalization it's true.