Traditionally, crop yield has been the main focus of agricultural policies and technological interventions. While there have been continuous efforts to improve farming practices towards food and farm sustainability, it is necessary to develop a metric to assess farming system in a holistic manner. In order to address this need, we have identified a holistic set of indicators covering long-term environmental impacts on agro-ecosystem, and socio-economic sustainability of farmers. These indicators are classified under three widely accepted dimensions: economic, social and ecological dimension. These dimensional indices are further aggregated to give a single holistic index called Farm Assessment Index (FAI). The weightages for economic (40.5%), social (28.5%) and ecological (31%) dimension were chosen by an expert panel consisting of two bureaucrats, two scientists, one academician, and one member representing non-governmental organizations. The methodology was applied to evaluate farming practices of a set of 100 organic and 100 chemical farmers, across Maharashtra, Tamil Nadu, Odisha and Karnataka. While there have been variations in yield and income trends, FAI score of most organic farms is better than the corresponding chemical farms. Even in the cases where the gross income from chemical farms is relatively higher, the economic index is higher for organic farms due to their higher benefit-cost ratio, lower risk, as well as better resource use efficiency. Similarly, in case of the social and environmental index, organic farms have scored higher than chemical farms due to the impacts caused by excessive fertilizer and pesticide usage in chemical farms. The variance of FAI among chemical farms was significantly higher than that of organic farms. Further, among the chemical farms, less intensive crops like wheat and gram have significantly higher index score than that of input-intensive cotton cultivation. The comparative studies have shown that the organic farming practices needs to be encouraged for improving the long-term socio-economic viability of the farmers and ecological sustainability of agriculture.