Large Scale Machine Learning

Large Scale Machine Learning is different from traditional machine learning in the sense that it involves large data - which is data that may have a very high number of training/test instances, features or classes.

There are a couple of issues which are to be kept in mind when handling with data of such high dimensionality (billions), most prominent of them being scalability. Recent research in the area is on methods which utilize tree-based or other O(log n) approaches (exploiting hierachy present in the data) which reduce training and/or prediction time.

The rise of Big Data requires complex Machine Learning models with millions to billions of parameters, that promise adequate capacity to digest massive datasets and offer powerful predictive analytics (such as high-dimensional latent features, intermediate representations, and decision functions) thereupon. In turn, this has led to new demands for Machine Learning (ML) systems to learn complex models with millions to billions of parameters. In order to support the computational needs of ML algorithms at such scales, an ML system often needs to operate on distributed clusters with 10s to 1000s of machines; however, implementing algorithms and writing systems softwares for such distributed clusters demands significant design and engineering effort. A recent and increasingly popular trend toward industrial-scale machine learning is to explore new principles and strategies for either highly specialized monolithic designs for large-scale vertical applications such as various distributed topic models or regression models, or flexible and easily programmable general purpose distributed ML platforms—- such as GraphLab based on vertex programming, and Petuum using parameter server. It has been recognized that, in addition to familiarity of distributed system architectures and programing, large scale ML systems can benefit greatly from ML-rooted statistical and algorithmic insights, which can lead to principles and strategies unique to distributed machine learning programs. These principles and strategies shed lights to the following key questions—- How to distribute an ML program over a cluster? How to bridge ML computation with inter-machine communication? How to perform such communication? What should be communicated between machines?—- and they span a broad continuum from application, to engineering, and to theoretical research and development of Big ML systems and architectures. The ultimate goal of large scale ML systems research is to understand how these principles and strategies can be made efficient, generally-applicable, and easy to program and deploy, while not forgetting that they should be supported with scientifically-validated correctness and scaling guarantees.

Advantage of Large-Scale Production:

The modern factory system, with its extensive use of machinery and division of labour, is responsible for large-scale production.

Economy of Specialized and Up-to-date Machinery:

There is a large scope for the use of machinery which results in lower costs. A large producer can install an up-to-date and expensive machinery. He can also have his own repairing arrangement. Specialized machinery can be employed for each job. The result is that production is very economical. A small producer with a small market cannot keep the machinery continuously working. Keeping it idle is uneconomical. A large producer can work it continuously and reap the resulting economies.

Economy of Labour:

In a big concern, there is ample scope for division of labour. Specialized labour produces a larger output and of better quality. It is only in a large business that every person can be put on the job that he can best perform. The large-scale producer thus gets the best out of every person he employs.


Economies of Bulk buying and selling:

While purchasing raw material and other accessories, a big business can secure specially favourable terms on account of its large custom. Whine selling its goods, it can attract customers by producing a greater variety and by ensuring prompt execution of orders. Even a small rate of profit results in larger sales and higher net profits in a large-scale business.

Economies of Overhead Charges:

The expenses of administration and distribution per unit of production in a big business are much less. Interest, the pay bill, and other overhead charges are the same whether production is large or small. Thus, the same amount of expenditure being distributed over a larger output results in a lower cost per unit.

Economy in Rent:

A large-scale producer makes a saving in rent too. If the same factory is made to produce a large quantity of goods, the same amount of rent is divided over a large output. This means that the cost per unit in respect of rent comes to a much smaller amount.

Experiments and Research:

A large concern can afford to spend liberally on research and experiments. It is well known that, in the long run, these expenses more than repay. Successful research may lead to the discovery of a cheaper process. This may bring a large profit. Only a large-scale business can incur such expenditure.

Advertisement and Salesmanship:


A big concern can afford to spend large amounts of money on advertisement and salesmanship. Ultimately they do bear fruit. Also, the amount of money spent on advertisement per unit comes to a low figure when production is on a large scale. The salesman can make a careful study of individual markets and thus acquire a hold on new markets or strengthen it on the old ones. Thus a large-scale producer has a greater competitive strength.

Utilization of By-products:

A big business will not have to throw away any of its by-products or waste products. It will be able to make an economical use of them. A small sugar factory has to throw away the molasses, whereas a big concern can turn it into power-alcohol. By utilising by-products, it can lower the cost of production.

Facing Adversity:

A big business can show better resistance in times of adversity. It has much larger resources. Losses can be easily borne. A small concern will simply collapse under such a strain.

Cheap Credit:

A large business can secure credit facilities at cheap rates. Its credit in the money market is high and the banks are only too willing to give advances. Low cost of credit reduces cost of production.

These are some of the advantages that a large-scale business has over a small-scale business. It can produce better goods at lower cost. But let us see the other side.

Disadvantages of Large-scale Production:

Large-scale production is not without its disadvantages.

Some of these disadvantages are:

Less Supervision:

A large-scale producer cannot pay full attention to every detail. Costs often rise on account of the dishonesty of employees or waste of material by them. This is due to the lack of supervision. Owing to laxity of control, costs of production will go up.

Individual Tastes Ignored:

Large-scale production is a mass production or standardised production. Goods of uniform quality are turned out irrespec¬tive of the requirements of individual customers. Individual tastes are not, therefore, satisfied. This results in a loss of customers.

Absence of Personal Element:

A large scale business is generally managed by paid employees. The owner is usually absent. The sympathy and personal touch, which ought to exist between the master and the men, are missing, Frequent misunderstandings lead to strikes and lockouts. This is positively harmful to the business.

Possibility of Depression:

Large-scale production may result in over¬production. Production may exceed demand and cause depression and unem¬ployment. It is not always easy or profitable to dispose of a large output.

Dependence on Foreign Markets:

A large-scale producer has generally to depend on foreign markets. The foreign markets may be cut off by war or some other upheaval. This makes the business risky.

Cut-throat Competition:

Large-scale producers must fight for mar¬kets. There is wasteful competition which does no good to society or to businessmen. Many promising businesses are ruined.

International Complications and War:

When the large-scale producers operate on an international scale, their interests clash either on the score of markets or of materials. These complications sometimes lead to armed conflicts. Many modern wars arose on account of scramble for materials and markets.

Less Adaptability:

A large-scale producing unit finds it very difficult to switch on from one type of production to another. In a depression, small-scale firms move away from declining trades to flourishing ones easily. In this way they are able to avoid losses. This adaptability is lacking in a big business.