At its creation in 1968 the San Rafael Wilderness consisted of only . Originally it was the San Rafael Primitive Area. Due to a disagreement between the Forest Service, US Congress and conservationists over of natural grass openings called "portreros", which contained pictographs from the Chumash Indians, it took a long time for the wilderness designation. In 1992, after the passage of the Los Padres Condor Range and River Protection Act, Congress added an additional adjacent to the original area on the northwest. The wilderness is also adjacent to the Dick Smith Wilderness to the east; this protected area was created in 1984.