The chapel occupies the site of a medieval settlement first recorded in written sources in 1337, where an earlier wooden church around 14 metres long once stood. By the mid-18th century that building, along with the neighbouring wooden church at Målilla, had fallen into disrepair. Following a 1768 episcopal visitation both were condemned as unsound.
Since the two parishes lay only a short distance apart, it was decided to build a single shared church at Målilla rather than repair both. The Gårdveda parishioners appear to have had second thoughts about the plan and were reluctant to contribute building stone, and it took until October 1800 for royal approval of the parish union to be granted. Various designs for the new joint church were considered, including round and octagonal plans, before the church eventually built at Målilla in 1820–22 took its final form, intended to seat some 1,750 people.
Once the new church was complete, the old Gårdveda church was sold at auction in 1822. One of its two bells was reportedly sold on to Persnäs on Öland, where it is said to remain. Little else survived: the foundations of the old church can still be seen beside the chapel, marked by a memorial cross erected to the south.
The present chapel, built in 1960, thus serves a churchyard with a much longer history than the building itself. Gårdveda kyrkogård remains in use today, linking the modern chapel to the long-abandoned medieval parish it replaced.