-Sewage sanitation and environmental protection
-Federal Reserve expenses
In 2010, government spending was 40% of GDP and most of that spending is in the form of transfer payments like social security as well as government insurance schemes like medicare and medicaid. We can still have the public goods that the public sector provides without having a big government.
There are also economic reasons for having small government. According to a 2011 survey of the academic literature on the relationship between government spending and economic growth found that, "The most recent studies find a significant negative correlation [between government size and economic growth]: an increase in government size by 10 percentage points is associated with a 0.5% to 1% lower annual growth rate." 
Many studies have tried to find the size of government which maximizes economic growth. The findings are almost unanimously within the 15%-25% of GDP range. For example, researchers at the Institute for Market Economics found:
"The evidence indicates that the optimum size of government, e.g. the share of overall government spending that maximizes economic growth, is no greater than 25% of GDP based on data from the OECD countries. In addition, the evidence indicates that the optimum level of government consumption on final goods and services as a share of GDP is 10.4% based on a panel data of 81 countries. However, due to model and data limitations, it is probable that the results are biased upwards, and the “true” optimum government level is even smaller than the existing empirical study indicates." 
The evidence appears to suggest that the economy grows faster when government is smaller, thus providing support for the classical liberal viewpoint that government is needed to provide public goods, but is detrimental to growth once it starts crowding out spending and investment in the private sector via borrowing or taxation in order to pay for things like the welfare state.