First established by miners after the discovery of gold July 4, 1862, the community was originally called Independence and was situated about a mile-and-half west-north-west of the present location. The name Granite was chosen when its post office was established in 1878, the name Independence already being used in Polk County, Oregon.

A. G. Tabor, who had staked the first mining claim on Granite Creek (the Independence) and was the only merchant in town at the time, served as the first postmaster. The city of Granite was incorporated in April 1900 with Grant Thornburg as mayor.

With more than eighty percent of the men in the area directly employed in the mining industry, the viability of the area’s economy practically collapsed when gold mining was made illegal in 1942 by Federal Public Law L-208, under the authority of the War Labor Act. The city had a population of 45 in 1930, 86 in 1940, 40 in 1950, and two in 1960. Although electrical service, provided by a local power plant on Clear Creek, and telegraph and later telephone lines were operational in the early days, they fell out of use and were abandoned after World War II. Modern telephone service was extended to Granite in the year 2000.

As of the 2000 census, the city had a total population of 24, up from its population of 10 in 1990. It is the second-smallest (according to population) incorporated city in Oregon, after nearby Greenhorn, which has a population of zero.