The church that stands today replaced a Romanesque sandstone building dating from the late 1500s. That earlier structure had fallen into such poor condition that its west gable collapsed during a service, according to one source in 1855 and, according to another, only implied as being some point before the rebuilding; while parishioners waited for a new church to be built, services were held at nearby Hällums kyrka, which was demolished once Skarstad's new building was complete.
Sources differ on who designed the replacement: one account credits the architect Johan Adolf Hawerman, while the Wikipedia entry attributes the 1859 building to Johan Fredrik Åbom. The exterior is built in a Gothic Revival style with a stone structure comprising a nave, an eastern chancel, a slightly recessed west tower and a narrower three-sided sacristy to the east of the chancel. Inside, the character is calmer and more classical, centred on a monumental altar wall shaped like a triumphal arch framing a cross with a shroud.
Among the church's furnishings are two fonts of very different ages: one carved from limestone in 1662, and an older one dating from the late 1200s. A seated wooden Madonna figure from the 1200s, carved in oak and standing 67 centimetres tall, was also associated with the church, though it is now kept at the Gothenburg City Museum rather than on site. The church's large bell is of a late medieval type and bears no inscription.
The organ, on the west gallery, dates from 1867 and was built by Johan Nikolaus Söderling, with both the instrument and its sounding façade from that period. It was rebuilt in 1949 and further renovated and enlarged in 1974, both times by the firm Nordfors & Co.
The churchyard's oldest sections are thought to be roughly contemporary with the medieval church, and the earliest legible gravestone dates from the 1700s. A well-known avenue of Swedish birch trees leading from the western entrance to the church was planted in 1959.