The original church on this site was a small medieval stone building, probably dating from the 1300s, dedicated to Saint Olof. A vicar in Gryt is recorded as early as 1423, though little else is known of the church's early history. By the 17th and 18th centuries it had a rectangular plan with a straight-ended chancel, a north sacristy and a porch. Unusually, over the western gate of the churchyard wall stood a stone structure resembling an outdoor pulpit, with steps and a small platform, from which priests are said to have sprinkled holy water on churchgoers; this opening was walled up in 1715.
When the old church was demolished in 1798, its sacristy was spared and later converted into a small chapel, now known as St Olof's Chapel. It has served various purposes over the centuries, including as a storehouse and parish granary, before being remodelled as a chapel in the early 1900s. It is still used for services and church occasions, particularly in winter, though current access arrangements are not covered here.```
The present church was built in 1798–1800 to designs begun by county master builder Casper Seurling and completed under builder Berg of Fogelvik, in a restrained neoclassical style typical of a large hall church of the period. It has a rectangular nave with a three-sided chancel and a tower with a lantern-topped spire. The roof was originally shingled but had to be replaced with tiles within a few years owing to the shallow pitch.
Inside, the dominant feature is the large altarpiece depicting the Ascension of Christ, painted in 1802 by Pehr Hörberg, one of the most prolific Swedish church painters of his day. The pulpit dates from 1875, though it has been moved more than once during later restorations, at times standing near the sacristy entrance. The red granite font was added in 1925, and a votive ship, made by seaman Manfred Pettersson, was hung in the church in 1955.
The organ, built in 1835 by Pehr Zacharias Strand of Stockholm, has sixteen stops and remains a notable example of the instrument-maker's work. The composer Ture Rangström is buried in the churchyard.