By the 1860s, Little Italy mostly located on Telegraph Hill was considered to be the model Italian immigrant community, free of the prejudice, crime, and poverty which had become synonymous with Little Italys in Eastern cities. The prominent inhabitants claimed this was because Little Italy had an 80 percent northern Italian population, in contrast to the southern Italian dominated Eastern Seaboard. The colony was organized around the St. Francis of Assisi Church, and was dominated by immigrants from similar villages in Italy who had transplanted themselves in San Francisco. The northern Italians were not Telegraph Hill's only inhabitants, however. Irish immigrants had arrived in San Francisco earlier, and lived higher up on the hill. There was violence between the two groups, mostly between young males. By the turn o f the century, however, Italians outnumbered Irish, and most of the Irish left after the 1906 earthquake.
But the Italian hold on Telegraph Hill was far from secure. Quarrymen, of whom the brothers George and Harry Gray were the most prominent, were blasting the east face of the hill away with dynamite. The city ordered the quarrymen to halt their blasting, and the inhabitants of all heritages would often pelt the quarry workers with rocks. But it was not until World War I, through the efforts of predominantly Progressive women's organizations that the city ended all blasting and quarrying. At the same time, Little Italy was expanding into other parts of the city.